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Overview 



Improving the success of academically underprepared students who are in need of developmental (or 
remedial) education is a key challenge facing community colleges today. Many of these students 
enter college with little awareness of these institutions’ expectations or a clear model for how to 
make effective decisions about their academic careers. To help students address these challenges, a 
number of colleges across the country have looked to success courses (also called study skills, 
student development, or new student orientation courses). This report analyzes a success course for 
developmental education students at Guilford Technical Community College in Greensboro, North 
Carolina, and its impact on students’ psychosocial s ki lls and behaviors and academic achievement. 

After joining Achieving the Dream: Community Colleges Count in 2004, a national organization 
designed to mentor colleges through an institutionwide, student success-oriented improvement 
process, Guilford chose to offer a revised version of its student success course to developmental 
education students, aimed at improving psychosocial awareness and academic achievement. 
Modeled on Skip Downing’s On Course philosophy and curriculum, it placed an intensive focus on 
changing students’ behaviors and attitudes, including increasing their awareness of their and others’ 
emotions, understanding their own learning styles, improving time management skills, and recogniz- 
ing their responsibility for their own learning. Guilford hoped that these changes in students’ 
personal habits and behaviors might help them take better control of their academic lives, which 
would ultimately result in gains in achievement. 

This study employed random assignment methodology to examine the impact of Guilford’s success 
course. The key findings presented in this report are: 

• Guilford’s implementation of its student success course stayed true to the On Course philoso- 
phy, with a strong emphasis on improving students’ psychosocial sk il ls and habits. 

• Challenges emerged during the study in maintaining instructors’ enthusiasm for teaching the 
course. 

• The course had a positive impact on students’ self-management, interdependence, self- 
awareness, interest in lifelong learning, emotional intelligence, and engagement in college 
among students with low levels of these attributes. 

• But the gains in efficacy did not lead to meaningful effects on students’ academic achievement 
during the program semester or in postprogram semesters. Despite the absence of an overall 
effect, the program did have positive effects on the first cohort of students enrolled in the study, 
with students demonstrating improved grades, retention in college, and credits earned. 

The results of this study reveal that improvements in students’ attitudes and behaviors may not 
necessarily translate easily into better academic outcomes, though the strength of program imple- 
mentation may play an important role in these effects. Additionally, the program’s limited effects 
suggest that community colleges should look to more comprehensive ways of improving develop- 
mental education students’ academic achievement, including reforms in developmental education 
instruction. 




Contents 



Overview iii 

List of Exhibits vii 

Preface ix 

Acknowledgments xi 

Executive Summary ES-1 

Chapter 

1 Introduction 1 

Background on Student Success Courses 2 

Overview of Guilford’s Student Success Course 5 

Overview of the Evaluation 5 

Contents of This Report 6 

2 The College, the Study Sample, and Data Sources 7 

Students Targeted hy the Program 7 

Random Assignment Methodology 9 

Characteristics of the Sample 10 

Data Sources and Eollow-Up Periods 10 

Conclusion 13 

3 Implementation of the Student Success Course 15 

Implementation of the On Course Philosophy in Guilford’s Student Success Course 15 

A Success Course for Developmental Education Students 17 

Staffing and Training 18 

Participation in the Student Success Course 19 

Instruction and Learning in the Student Success Course 22 

Contrast between Program Group Students’ and Control Group Students’ Experiences 23 

Conclusion 25 

4 Impacts of the Student Success Course 27 

Effects on Students’ Socioemotional Skills and Well-Being 28 

Academic Bindings 3 1 

Subgroup Analysis: Effects on Students Enrolled in Courses During the Program 

Semester 35 

Subgroup Analysis: Effects on Students by Cohort 37 

5 Summary and Conclusions 45 




Appendix 



A Technical Appendix 49 

B Sensitivity Analysis 55 

C Survey Outcomes by Cohort 61 

References 65 

Earlier Publications on Achieving the Dream 67 



VI 




List of Exhibits 



Table 

2. 1 Selected Characteristics of Students Enrolled, Fall 2008 8 

2.2 Characteristics of Sample Members at Baseline 1 1 

3. 1 Program Requirements/Participation 20 

4. 1 Survey Outcomes 29 

4.2 Academic Outcomes in Program Semester 32 

4.3 Academic Outcomes in First Postprogram Semester 34 

4.4 Academic Outcomes: Cumulative from Program to Third Postprogram 

Semester 36 

4.5 Transcript Outcomes, Program Semester, hy Cohort 38 

4.6 Transcript Outcomes, First Postprogram Semester, by Cohort 40 

4.7 Cumulative Academic Outcomes, Program to Third Postprogram Semester, by 

Cohort 42 

B. 1 Academic Outcomes in Program Semester (Program Semester Enrollees 

Only) 58 

B. 2 Academic Outcomes in First Postprogram Semester (Program Semester 

Enrollees Only) 59 

C. 1 Survey Outcomes, by Cohort 63 

Box 

3. 1 Core Principles of the On Course Curriculum 17 

3.2 Flow to Read Tables in This Report 21 

3.3 Differences in Services Provided to Program and Control Group Students 24 



vii 




Preface 



Community colleges have long been a crucial pathway into the middle class for low-income 
students who face barriers to education. However, these institutions also face a number of 
challenges in helping their students succeed, including historically low graduation rates and 
large numbers of students who enter college academically underprepared. The majority of 
community college students are often required to take a lengthy series of developmental, or 
remedial, education classes. Many never successfully progress through preparatory classes into 
a college-level curriculum. 

Given these challenges, many community colleges are experimenting with new meth- 
ods for improving developmental education students’ outcomes. Student success courses — 
also known as student development, study skills, or student orientation courses — are one 
popular method colleges have used to try to improve students’ understanding of college and 
their ability to navigate their way through school. This study, which analyzes the impacts of a 
student success course on developmental education students’ achievement at Guilford Technical 
Community College, provides one look at how these courses affect students’ achievement. 

The findings from this study reveal that Guilford’s success course was able to foster 
some changes in students’ attitudes and perspectives, such as their self-management, self- 
awareness, interest in lifelong learning, and engagement in college. Unfortunately, these 
improvements did not translate into improved academic outcomes for the overall group of 
students to whom the course was offered. 

Although the positive effects on the first cohort’s academic achievement are intriguing, 
the absence of an overall effect lends a more critical perspective on student success courses than 
have other recent quasi-experimental and experimental studies. While these courses have had a 
modest effect on student outcomes when paired with tutoring (as in Chaffey College’s Opening 
Doors program) or in learning communities (as at Kingsborough Community College), this 
study suggests that they may have more limited value when offered without such supports. 

While success courses may provide a positive benefit to students’ understanding of col- 
lege and its expectations, such courses may need to be more limited or integrated within larger 
stmctural changes in developmental education to improve students’ academic progress. 

Gordon L. Berlin 
President 
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Executive Summary 



Thousands of community college students across the country confront serious questions every 
day about their course-taking and career choices. They often are required to make key decisions 
relatively quickly, often with little introduction to how to make these choices most effectively. 
To further complicate matters, most community college students enter college academically 
underprepared and are required to take developmental, or remedial, English, math, and/or 
reading courses in order to advance to a college-level curriculum. ' 

Many colleges have looked to success courses,^ which seek to orient students to college 
life and assist them in making important decisions about college and their careers, as a way to 
help students address these challenges. In 2005, Guilford Technical Community College in 
Greensboro, North Carolina, joined many colleges across the nation in developing a success 
course for developmental education students as one of their strategies under the auspices of the 
Achieving the Dream: Community Colleges Count initiative. Achieving the Dream is a national 
organization designed to mentor colleges through an institutionwide, data-driven, student 
success-oriented improvement process.^ Along with developing students’ study habits, Guil- 
ford’s student success course placed an intensive focus on improving students’ psychosocial 
and soft skills, such as their awareness of their and others’ emotions, understanding of their own 
learning style, improved time management, and a recognition of their responsibility for their 
own learning. Guilford hoped that changing students’ personal habits and attitudes might help 
them take better control of their academic lives and improve their overall achievement. 

This report analyzes the impact of Guilford’s success course on developmental educa- 
tion students’ academic achievement as well as several social and psychological measures, 
including motivation, self-concept, and commitment to college. In sum, this study found that the 
success course had few effects on students’ academic achievement, though the evidence 
suggests that there were some positive, differential effects for students in the first group to enter 
the study. The success course was also found to have a positive impact on several psychosocial 
outcomes, including students’ self-management, interdependence, self-awareness, interest in 
lifelong learning, emotional intelligence, belief in self, and positive engagement in college. 



' Adelman (2004); Attewell, Lavin, Domina, and Levey (2006); Bailey, Joeng, and Cho (2010). 
^These courses are also termed study skills, student development, or new student orientation courses. 
^Achieving the Dream (2009). 
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Guilford’s Student Success Course 



Student success courses have become a popular strategy for increasing students’ achievement in 
community colleges throughout the country/ Guilford was particularly drawn to improving its 
developmental education students’ success rate after noting their poor academic performance in 
the college’s Achieving the Dream data analyses. In order to better address students’ socioemo- 
tional needs as well as their academic skills, Guilford chose to redesign an existing study skills 
course, shifting the course toward a greater focus on students’ responsibility for their own 
learning. Modeled on Skip Downing’s On Course philosophy,^ this newly revised course 
focused on helping students overcome their personal challenges through intensive reflection on 
their past history and future goals. Additionally, the course provided some opportunities for 
academic skill-building through class presentations, journal writing, quizzes, and an end-of- 
semester course project. The course was offered as a two-credit class to students in need of one 
or more developmental education course in reading, English, or math and was taught by faculty 
or staff members trained in the On Course philosophy and pedagogy. 

MDRC’s evaluation of Guilford’s success course began in spring 2008. It employs a 
random assignment evaluation design to examine the impact of the program on students’ 
achievement. A total of 91 1 students were enrolled in the study over the course of three semes- 
ters (spring 2008, fall 2008, and spring 2009). Using a lottery-like process, students were 
assigned to either a program group (458 students), which received the success course interven- 
tion, or to a control group (453 students), which received the college’s regular services (and 
were not enrolled in the success course). By comparing the outcomes of program and control 
group students over time, the study is able to gauge the impact of Guilford’s student success 
course on academic measures such as students’ persistence in college, grades, and course 
completion. Quantitative data on students’ backgrounds and academic achievement were also 
collected through a baseline information form and student transcript data. Qualitative data were 
gathered on the program and students’ experiences during two site visits to the college. During 
these visits, researchers interviewed administrators, faculty, staff, and participating students and 
conducted a student survey, which asked questions about the personal beliefs and habits the 
success course was expected to affect. 



Key Findings 

• Guilford’s implementation of the student success course stayed true to 
the On Course philosophy, with a strong emphasis on improving stu- 



"^Stovall (2000); Derby and Smith (2004); Zeidenberg, Jenkins, and Calcagno (2007); Scrivener, Sommo, 
and Collado (2009); Scrivener et al. (2008). 

^Downing (2008). 
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dents’ psychosocial s kil l s and habits. Instructors in the course were gener- 
ally well trained; all faculty and staff received at least a three-day introduc- 
tion to the On Course pedagogy and curriculum. Lessons tended to engage 
students in critical reflection about their own personal experiences and habits, 
with an effort to promote an awareness of their own role in their learning and 
future responsibilities. Students were encouraged to reflect upon their role in 
their learning through assignments such as weekly journal writing or by shar- 
ing their personal experiences with the class. Students also received some in- 
stmction and practice in study and academic skills through activities such as 
quizzes and a formal class paper and presentation. 

• Evidence suggests that the success course was more strongly implement- 
ed in the first semester of the study than in later semesters. During the 
first semester of the course, instructors had a great deal of enthusiasm for 
teaching the course and met monthly to discuss their teaching and best prac- 
tices. During the final two semesters, however, this enthusiasm was less ap- 
parent. Several new instmctors noted that they felt less prepared to teach the 
curriculum and deal with students’ socioemotional issues, while some sea- 
soned instmctors explained that they had seen few benefits from the course 
in their first semester of teaching it. Additionally, because monthly meetings 
were discontinued in the final two semesters, several new instmctors felt that 
they received less support in implementing the course. 

• Students’ participation in Guilford’s success course declined over the 
course of the program semester, resulting in only 61 percent of the pro- 
gram group students taking and completing the course. However, it is 
difficult to conclude how this may have affected students’ outcomes. 

Weeks or months often ensued between the time of random assignment and 
registration and the first day of classes, giving students ample time to reor- 
ganize their schedules and drop the class. When conducting exploratory 
analyses, the effect of student participation on academic outcomes was in- 
consistent over time. As a result, it is unclear whether a higher participation 
rate would have made the program more successful in improving academic 
outcomes. 

• A follow-up survey of program and control group students found that 
Guilford’s student success course had a positive impact on students’ self- 
management, interdependence, self-awareness, interest in lifelong learn- 
ing, emotional intelligence, and positive engagement in college among 
students with low levels of these attributes. This suggests that Guilford’s 
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success course was able to improve personal decision making, awareness of 
themselves and others, and commitment to their education among students 
who had lower levels of these abilities. 

• Guilford’s success course had no meaningful effects on students’ aca- 
demic achievement during the semester that the program operated or in 
the following semesters. There were no statistically significant differences 
between control and program group students in course registration or enroll- 
ment, credits attempted, course pass rates, course withdrawal rates, credits 
earned, successful completion of developmental education courses, or grade 
point averages, either during the semester of the success course or in the 
three following semesters. This lack of meaningful academic impacts sug- 
gests that the social and psychological impacts, though significant, may not 
have been strong enough to affect students’ achievement and that other pro- 
grams and services may be needed to improve their academic performance. 

The results of this study reveal that Guilford’s new success course, with its focus on so- 
cial and emotional skills, was able to foster some changes in students’ attitudes and perspec- 
tives. Unfortunately, however, these improvements did not translate into improved academic 
outcomes for the overall group of students to whom it was offered. While the mixed results 
from this study do not provide strong evidence that a success course alone improves student 
academic outcomes, these findings should be taken in concert with other studies that have 
shown more positive results for these courses, particularly when success courses are one 
component in more comprehensive programs. 

For instance, the findings in this study are not unlike those from a student success 
program at Chaffey College, which targeted probationary students. At Chaffey, an enhanced 
version of the student success program, which framed the program as mandatory and included 
visits to the college’s campus-based tutoring centers in reading, writing, and math, had 
positive impacts on students’ engagement and achievement.® Similarly, a number of positive 
impacts on students’ outcomes have been observed with learning communities that included a 
student success course. At Houston Community College, for example, a program that linked a 
success course and developmental math resulted in positive effects on students’ developmen- 
tal math pass rates.’ 

However, even when a program is well implemented, the impact of a one-semester in- 
tervention on students’ achievement may be modest and less likely to achieve the lofty goals of 



®Scrivener, Sommo, and Collado (2009). 
’Weissman et al. (201 1). 
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improved grades, credits earned, and graduation across the board. In each of the studies noted 
above, impacts on students’ academic achievement tended to be modest and generally sustained 
only during the semester in which the program operated or one semester after. These short-term 
effects suggest that other issues may be creating more substantial barriers to students’ academic 
progress than those a success course can address. For instance, while success courses may help 
improve students’ soft skills and give them a better understanding of how to manage college 
life, students may still be overwhelmed by larger challenges in their lives, such as an inability to 
afford their school tuition or the struggle to balance work, school, and family responsibilities. 
Similarly, success courses do not address some of the stmctural and academic barriers students 
may face in college, such as the need to pass a long sequence of developmental education 
courses or continued failure in a particular academic course. 

Success courses can play an important role in helping students to learn about the variety 
of programs and services that may benefit them during their academic career. However, these 
courses also present an opportunity cost for students, as they often enroll in a success course at 
the expense of taking an additional academic course. As this study found, students acquired 
fewer academic credits during the semester that they took the two-credit success course, in part 
because they were unable to fit in an additional academic course. An additional concern is that 
credits for most success courses are not transferrable to four-year institutions. 

With these concerns in mind, colleges may wish to consider ways in which success 
courses can be incorporated into larger, more systemic approaches to improving developmental 
education students’ academic experiences. Given the relatively short-term effects of these 
courses and the opportunity costs they present to students, colleges may need to look toward 
less ambitious interventions that support students’ psychosocial well-being but do not require 
them to choose between academic courses and their introduction to college life. In order to 
effect greater changes in students’ achievement, colleges might consider more comprehensive 
approaches to improving students’ academic performance, such as reforms in developmental 
education instmction or the stmcturing of developmental education course sequences. A 
different combination of these efforts, which address students’ social challenges while also 
focusing more concretely on their academic needs, may prove even more promising. 
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Chapter 1 

Introduction 



Imagine your first day of college, a day when you are excited about your future prospects and 
nervous about your ability to meet the demands of a new school. A book sits before you, a text 
that could determine whether or not you succeed in college. However, rather than arithmetic 
formulas or philosophical treatises, this book contains a different set of challenges. It sets forth a 
bewildering array of courses, registration procedures, and degree requirements, asking you to 
make choices with which you have little experience. How do you decide which courses to take 
and how to focus your studies? On what basis should you make these important decisions about 
your future at college and in your career? As the semester goes on, an even more daunting array 
of questions faces you: How can you balance your studies with your responsibilities outside of 
school? What can you do if you fall behind in a particular subject? And most important, to 
whom can you turn for answers? 

Thousands of students confront questions like these as they navigate their first semes- 
ters at community colleges across the country. Regardless of whether they are fresh from high 
school or returning after years away from education, these new students must make critical 
decisions about their course work and their careers relatively quickly, often with little guidance. 
To further complicate matters, the majority of community college students enter college 
academically underprepared and are required to take developmental, or remedial, English, math, 
and/or reading courses in order to advance to a college-level curriculum. ' These challenges can 
be important impediments to students’ integration into college. 

In 2005, Guilford Technical Community College in Greensboro, North Carolina, joined 
a steady stream of colleges that are looking to student success courses as a way to assist new 
students in navigating their college careers.^ Often termed study skills, student development, or 
new student orientation courses, many colleges have implemented these courses in an effort to 
develop students’ study habits, awareness of their learning styles, and time management and 
test-taking skills. Generally offered as semester-long, credit-bearing courses, student success 
courses are often used as a way to build students’ academic and “soft skills,’’ help them learn 
about the college’s services, and orient them to the types of decisions and responsibilities they 
will face as college students. Many colleges have also aimed these courses directly toward 



' Adelman (2004); Attewell, Lavin, Domina, and Levey (2006); Bailey, Joeng, and Cho (2010). 
^Zachry and Qrr (2009). 
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students who take developmental education classes as a way to help these students get a leg up 
on their academic success? 

Developed as one of its strategies for the Achieving the Dream: Community Colleges 
Count initiative, Guilford’s student success course for developmental education students aimed 
to orient students to college life and assist them in making important decisions about college 
and their careers? This report analyzes the impact of Guilford’s success course on students’ 
academic achievement as well as several social and psychological measures, including motiva- 
tion, self-concept, and commitment to college. In sum, this study found that the success course 
had a positive impact on several psychosocial outcomes, including students’ self-management, 
interdependence, self-awareness, interest in lifelong learning, emotional intelligence, and 
positive engagement in college. However, though there were some positive effects on achieve- 
ment for students in the earliest cohort of the study, overall, the course had few effects on 
students’ academic achievement. This lack of meaningful academic impacts suggests that the 
social and psychological impacts, though significant, may not have been strong enough to affect 
students’ achievement and that other programs and services may be needed to improve their 
academic performance. 



Background on Student Success Courses 

Student success courses have become a popular strategy for increasing students’ achievement in 
community colleges throughout the country. A number of studies of individual programs in the 
early 2000s associated these courses with promising increases in students’ academic achieve- 
ment and persistence in college. For instance, participants in a student success course at a mral 
Midwestern community college earned higher first-term grade point averages, completed more 
credit hours, were more likely to persist in college, and had greater odds of graduating by the 
end of a three-year follow-up period than their nonparticipating counterparts.^ Similarly, a study 
of a semester-long student orientation course at a Midwestern community college found that the 
course was associated with completion of a degree, persistence, and reenrollment after taking 
short breaks from school.® Although neither of these studies employed random assignment 
methodology, which is considered the most reliable of program evaluation methods, their 
results are encouraging and suggest that more rigorous evaluation of success courses would be 
worthwhile. 



^Zeidenberg, Jenkins, and Calcagno (2007). 

"^Achieving the Dream: Community Colleges Count is now an independent national nonprofit organization. 
^Stovall (2000). 

®Derby and Smith (2004). 
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More recent studies have also found promising student achievement outcomes. For ex- 
ample, a quasi-experimental analysis of student success courses in Florida community colleges 
found that enrollment in these courses was associated with greater likelihood of earning a 
credential, staying in school, and transferring to Florida’s four-year colleges.’ Additionally, 
several random assignment studies have found that student success courses can have promising 
effects on students’ academic achievement and persistence in school when they are a part of 
larger packages of services. For instance, a student success program targeting probationary 
students at Chaffey Community College found that an enhanced version of the program, which 
framed the program as mandatory and included visits to the college’s campus-based tutoring 
centers in reading, writing, and math, had positive impacts on students’ engagement and 
achievement. After implementing the mandatory policy, the program had much higher partici- 
pation rates, resulted in positive impacts on students’ grade point averages, and was successful 
at moving students off probation.® Similarly, a learning communities program for entering 
freshman at Kingsborough Community College, which linked a student success course with 
English and content courses, found that students who participated in the program moved more 
quickly through their developmental English requirements and were more likely to pass a 
standardized reading and writing test needed to transfer to a four-year college.'^ However, both 
the Chaffey and Kingsborough success courses were integrated with other program supports, 
such as additional tutoring (Chaffey) or learning communities (Kingsborough), making it 
difficult to disentangle the effects of the success course alone on students’ outcomes. 

Promising results like these have led a number of individual schools, community college 
systems, and even states to mandate student success courses for newly entering students. Eor 
instance, the Virginia Community College System requires that all students at its 23 community 
colleges complete a Student Development Course within their first 15 credit hours.'® Similarly, 
first-time students at any of the Houston Community College System’s six campuses are 
required to take a Ereshman Success Course if they have earned 12 or fewer semester credit 
hours." Additionally, some community colleges require student success courses for developmen- 
tal students, in particular. Eor example, 12 of 28 Elorida community colleges require students 
who are taking one or more developmental courses to enroll in Student Eife Skills courses.'’ 

Some positive effects have been seen for success courses targeted specifically to devel- 
opmental education students, though the effects have depended on the course pairing and have 



’Zeidenberg, Jenkins, and Calcagno (2007). 

® Scrivener, Sommo, and Collado (2009). 
^Scrivener et al. (2008). 

"’Virginia Community College System (201 1). 
"Houston Community College System (2010). 
'’Florida Department of Education (2005). 
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tended to dissipate over time. On the positive side, the quasi-experimental study analyzing the 
overall effects of success courses in Florida community colleges noted that developmental 
students who took a success course were more likely to earn a credential than students enrolled 
in developmental courses alone. Similarly, learning communities programs at Houston 
Community College and Queensborough Community College, which linked student success 
courses with developmental math courses, helped students attempt and pass their developmental 
math courses at higher rates during the semester that students participated in the linked cours- 
es.'"' However, when a success course was linked with a developmental reading course at 
Hillsborough Community College, few effects were found on students’ achievement.'^ Similar- 
ly, in each of the programs noted above, the effects on students’ achievement diminished in the 
semesters after the programs ended. 

While success courses have shown some promising results and have become a popular 
method to try to improve community college students’ achievement, few rigorous studies have 
analyzed the effects of these courses alone on students’ outcomes. MDRC has conducted a 
number of studies analyzing the effects of student success courses within the context of learning 
communities, where a cohort of students co-enroll in two or more linked courses during a 
semester, or alongside other enhanced services, such as intensive advising."' However, because 
the success courses are offered in combination with other academic and social services, it is 
difficult to know what effect, if any, the success course itself had on students’ outcomes. A key 
question remains about whether success courses, which are generally designed to improve 
students’ study skills and introduce them to college expectations, are a strong enough interven- 
tion in and of themselves to have an impact on students’ academic achievement. 

This study helps answer this question by studying the effects of a success course alone 
on students’ outcomes. As will be discussed below, Guilford’s success course was designed to 
enhance students’ socioemotional skills, such as their emotional awareness, sense of self- 
responsibility, and self-management, while also focusing on improving some academic habits, 
such as studying, writing, and presenting in front of others. By analyzing the effects of this 
course in the absence of other enhanced services, this study is able to examine whether a one- 
semester, two-credit course, focused on both personal and academic abilities, translates into 
overarching changes in students’ academic performance. 



'^Zeidenberg, Jenkins, and Calcagno (2007). 

'"'Weissman et al. (201 1). 

'^Weiss, Visher, and Wathington (2010). 

'^Scrivener, Sommo, and Collado (2009); Scrivener et al. (2008); Weissman et al. (2011); Weiss, Visher, 
and Wathington (2010). 
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Overview of Guilford’s Student Success Course 

Like many colleges across the country, Guilford was drawn to student success courses as a way 
to help struggling students overcome their academic and social challenges. When the college 
joined Achieving the Dream: Community Colleges Count, a national organization designed to 
mentor colleges through an institutionwide, student success-oriented improvement process, 
administrators noted the poor academic performance of students who were required to take 
developmental courses. The college chose to implement a number of interventions to reach 
these students, with a newly revamped student success course playing a primary role. 

In order to better address students’ socioemotional needs as well as their academic 
s kil ls, Guilford chose to redesign an existing study skills course, shifting the course toward a 
greater focus on students’ responsibility in their own learning. Modeled on Skip Downing’s On 
Course philosophy,'^ this newly revised course focused on helping students overcome their 
personal challenges through intensive reflection on their past history and future goals. Addition- 
ally, the course provided some opportunities for academic skill-building through class presenta- 
tions, journal writing, quizzes, and an end-of-semester course project. The course was offered as 
a two-credit course to students in need of one or more developmental education courses in 
reading, English, or math and was taught by faculty or staff members trained in the On Course 
philosophy and pedagogy. Chapter 3 provides more in-depth information about the design and 
implementation of the program. 



Overview of the Evaluation 

This study employs a random assignment methodology to analyze the effects of Guilford’s 
success course on students’ academic and socioemotional outcomes. Students who met the 
study’s eligibility requirements (see Chapter 2) and agreed to participate in the study were 
assigned, at random, to a program group, who received the college’s success course, or a control 
group, who received the college’s standard services (excluding the success course).'® The study 
then tracked students’ outcomes over the program semester (when the success course was 
offered to the program group) and the three semesters following to see if there were differences 
between the outcomes of students in the program group and those in the control group. A 
random assignment design ensures that differences in motivation and demographic characteris- 
tics do not bias the results of the study, as the program and control groups are similar at the time 



'^Downing (2008). 

'®Guilford had only a limited amount of resources available for the college success course, thus limiting 
the number of sections it could offer each semester. The lottery system that random assignment offered was 
seen as a more fair way to allow students to register for the course, as it allowed each student an equal 
opportunity for enrollment regardless of their motivation or the timing of their registration. 
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of entry into the study. As such, any differences in the program and control groups’ outcomes 
can be attributed with a high level of confidence to the impact of the success course. 

This study used four primary means of data collection and analysis: (1) an implementa- 
tion study, based on interviews and focus groups, which analyzed the design and rollout of the 
program at the college; (2) observations of student success courses; (3) an analysis of a student 
survey, which gathered information on the socioemotional characteristics of both the program 
and control group; and (4) an impact analysis of students’ achievement using transcript data 
from the college. The study is centrally concerned with changes in students’ achievement, such 
as their persistence in college, grades, and credits earned. However, it also analyzes whether any 
changes in students’ behaviors and beliefs occurred as a result of the program. 



Contents of This Report 

This report provides an analysis of both the implementation of Guilford’s student success 
course and its impact on students’ attitudes, personal habits, and academic achievement. 
Chapter 2 provides a brief description of the research design, including how students were 
identified and enrolled into the research sample, along with the characteristics of the sample. 
Chapter 3 discusses Guilford’s implementation of the course for its developmental education 
student population. Chapter 4 provides an analysis of the impact of the course on students’ 
academic achievement and college-going skills. Chapter 5 concludes with an integrative 
discussion of these findings. 



6 




Chapter 2 

The College, the Study Sample, and Data Sources 



Guilford Technical Community College is located in Guilford County, North Carolina, a mid- 
sized county in the northern part of the state. It is the only community college in the county and 
is nestled among six other four-year colleges and universities in the area.' With a total of 1 1,226 
students in 2008 (when this study began), the college is divided across three cities: Greensboro, 
Jamestown, and High Point.^ With an operating budget of over $86 million, Guilford employs 
approximately 1,600 full-time and part-time faculty and staff and is the third-largest community 
college in the state. ^ 

The student success course described in this report was offered at all three of Guilford’s 
campuses. Guilford is representative of the Greensboro area at large, with a diverse student 
body made up primarily of white and African-American students (see Table 2.1)."' Like many 
community colleges across the country, a large proportion of the students attend the college part 
time, and nearly half of the student body receives some sort of financial aid. Far fewer students 
graduate than Guilford’s leaders would like, though this low graduation rate differs little from 
that of other community colleges in the country.^ 



Students Targeted by the Program 

Based on an analysis of student outcomes, Guilford developed a number of criteria for the 
student population that would be targeted by the study. First and foremost, the college sought to 
recmit students who were required to take one or more developmental courses, as these students 
appeared to be the neediest and the least likely to succeed at the college. Additionally, given 
some of the challenges for entering students in making the transition to college, Guilford chose 
to focus primarily on new students, including those who had earned 20 or fewer credits at the 
college. Finally, in order to provide a fair test of the course, certain students were excluded from 
the study, such as those who had previously taken a success course or had been exposed to the 
On Course curriculum in other classes. 



'Guilford County (n.d.). 

^U.S. Department of Education (201 1); Guilford Technical Community College (201 1). 
^U.S. Department of Education (201 1); Guilford Technical Community College (201 1). 
"'U.S. Bureau of the Census (2010). 

^U.S. Department of Education (2011). 
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Achieving the Dream: Community Colleges Count 
Table 2.1 



Selected Characteristics of Students Enrolled, Fall 2008 
Guilford Technical Community College 



Characteristic 


Students 


Total enrollment (N) 


11,226 


Enrollment by gender (%) 


Male 


44.1 


Female 


55.9 


Enrollment by race/ethnicity (%) 


Hispanic 


3.4 


White 


50.4 


Black 


36.8 


Other 


9.4 


Students enrolled full time (%) 


54.2 


Students receiving some form of financial aid (%) 


62.7 


First-time fall-to-fall retention rate' (%) 


Full-time students 


56.0 


Part-time students 


37.0 


Overall graduation rate'* (%) 


14.5 



SOURCE: U.S. Department of Education, National Center for Education Statistics, Integrated Postsecondary 
Education Data System (IPEDS). 

NOTES: “Percentages do not add up to 100 percent due to rounding. 

’’Data are for first-time, full-time students seeking a certificate or degree. 

‘’Retention rates measure the percentage of first-time students beginning their programs in the fall of 2007 
who continue into the following fall term. 

‘’Graduation rates are calculated for full-time, first-time students who began their program in 2005. Students 
are considered to have graduated if they completed their program within 150 percent of the normal time for 
completion. 



In sum, students participating in the study were: 

• At least 18 years of age 

• Required to take one or more developmental courses 

• New students or continuing students with fewer than 20 credit hours and who 
were not transfers from another college 

• Not majoring in degree programs that required a student success course as part 
of the curriculum, such as business, architecture, or machining technology 







• Not previous participants in a student success course 

• Available to attend a student success course at the scheduled time 

All students who met these criteria were invited to participate and offered an opportuni- 
ty to voluntarily enroll in the study. Observations of the enrollment process revealed that study 
intake was conducted as intended and yielded a satisfactory sample for analyzing the effects of 
the student success course. However, as will be noted later, some challenges were posed by 
program group students who withdrew from the success course after their registration. 

In order to have sufficient power to detect the effects of the program, Guilford and 
MDRC sought to enroll 1,000 developmental education students into the study over the course 
of three semesters, beginning with the spring 2008 semester. Because Guilford had limited 
resources available for the college success course, it could offer only a limited number of 
sections each semester. The lottery system that random assignment employed was seen as a 
more fair way to allow students to register for the course, as they each had an equal opportunity 
to enroll, regardless of their motivation or the timing of their registration. After random assign- 
ment, students assigned to the program group were required to register for and take the semes- 
ter-long, two-credit student success course. Control students were barred from taking the course 
but received the normal services offered to the general student population. After three semesters 
of sample enrollment (spring 2008, fall 2008, spring 2009), Guilford achieved a total sample 
size of 911 students. Of these 911 students, 458 were assigned to the program group, which 
made them eligible to receive the college’s success course. The remaining 453 students were 
assigned to the control group; they received the college’s standard services. A total of 33 
success courses were offered as a part of this program over the course of the three semesters. 



Random Assignment Methodology 

As discussed in Chapter 1, this study uses random assignment in order to analyze the effects of 
Guilford’s success course on students’ outcomes. Seen as the gold standard in educational 
research, random assignment methodology is frequently compared to a coin toss or lottery 
system, whereby study participants are randomly assigned to either a program group or a 
nonprogram (control) group. Because each student has an equal chance of being placed into 
either group, the program and control group students should have similar characteristics at the 
outset of the study. This design helps protects against certain biases, such as differences in 
students’ levels of motivation, and helps ensure that the observed differences can be attributed 
to the program rather than to the characteristics of the students. 

However, there are some limitations to this methodology. Because random assignment 
involves only students who volunteer to participate, the sample could be qualitatively different 
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from the group of students who did not volunteer to participate. Additionally, this study specifi- 
cally analyzed the outcomes of developmental education students, so the findings cannot be 
generalized for the larger student population at Guilford. 



Characteristics of the Sampie 

As can be seen in Table 2.2, the study sample was similar to the overall Guilford student popula- 
tion in that it was largely female and racially diverse, although a higher proportion of female and 
African-American students were present in the study sample than in the general student popula- 
tion. Additionally, just over half of the study sample students were under 21 years of age, 
revealing that many of the students in the sample were recent high school graduates. 

The figures displayed in Table 2.2 also indicate some of the challenges that students in 
the sample faced. For instance, about a quarter of all sample members reported being the first 
person in their family to attend college. Additionally, over one-third of the students who 
responded to the question were in households receiving some form of government assistance, 
such as food stamps, unemployment or disability insurance, supplemental security income, cash 
assistance, or welfare. Just over half the sample members were working when they entered the 
study, while only about a quarter said that they were financially dependent on their parents. 

While, in general, the program and control group students were similar, there were a 
few significant differences between the two groups. For instance, while over 40 percent of the 
overall sample had graduated from high school in the previous year, significantly fewer pro- 
gram group students were recent graduates than control group students. Additionally, a higher 
proportion of program group members had a working computer in their home, an Internet 
connection, and access to a working car. Modest variation across treatment groups in students’ 
characteristics will occur by chance in random assignment. However, this does not indicate any 
systematic difference between research groups and does not bias the results. 



Data Sources and Follow-Up Periods 

Quantitative Data 

Students enrolling in the study completed a baseline information form, which collected 
certain demographic and background information. These data were used to describe the sample 
as well as to analyze the similarities and differences between students in the program and 
control groups. 

Guilford also provided transcript data on students enrolled in the study (both program 
group and control group students). These data provided information about students’ persistence 
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Achieving the Dream: Community Colleges Count 
Table 2.2 



Characteristics of Sample Members at Baseline 
Guilford Technical Community College 



Characteristic 


Full 

Sample 


Program 

Group 


Control 

Group 


Gender (%) 


Male 


31.2 


30.7 


31.6 


Female 


68.8 


69.3 


68.4 


Age (%) 


18-20 years old 


53.6 


50.0 


57.2 ** 


21-25 years old 


12.4 


14.4 


10.4 * 


26 - 30 years old 


10.2 


11.9 


8.5 * 


3 1 and older 


23.8 


23.7 


23.9 


Average age (years) 


25.3 


25.5 


25.1 


Marital status (%) 


Married 


12.2 


12.5 


11.9 


Unmarried 


71.7 


72.0 


71.3 


Missing 


16.1 


15.5 


16.8 


Race/ethnicity'* (%) 


Hispanic 


5.7 


5.6 


5.9 


White 


28.5 


28.1 


29.0 


Black 


58.5 


58.4 


58.6 


Other 


7.2 


7.9 


6.5 


Number of children (%) 


None 


64.0 


64.4 


63.6 


1 


12.4 


10.6 


14.1 


2 


12.9 


13.0 


12.9 


3 or more 


10.7 


12.0 


9.4 


Household receiving any of the following benefits’’ (%): 


Unemployment/Dislocated Worker Benefits 


5.5 


4.8 


6.2 


Supplemental Security Income (SSI) or disability 


7.8 


7.8 


7.8 


Cash assistance or welfare (TANF) 


2.3 


1.7 


2.9 


Food stamps 


23.3 


22.0 


24.6 


None of the above 


51.9 


54.1 


49.7 


Missing 


17.3 


16.3 


18.4 


Financially dependent on parents (%) 


27.7 


27.3 


28.0 


Missing (%) 


11.0 


9.4 


12.5 


Currently employed (%) 


54.3 


55.0 


53.5 



(continued) 
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Table 2.2 (continued) 



Characteristic 


Full 

Sample 


Program 

Group 


Control 

Group 


Date of high school graduation or GED receipt (%) 


During the past year 


45.8 


41.6 


50.1 ** 


Between 1 and 5 years ago 


19.1 


20.0 


18.2 


Between 5 and 10 years ago 


9.6 


11.1 


8.0 


More than 10 years ago 


25.5 


27.3 


23.8 


Student status (%) 


Incoming freshman 


82.9 


83.1 


82.7 


Returning student 


17.1 


16.9 


17.3 


Completed any college courses (%) 


19.9 


19.4 


20.5 


First person in family to attend college (%) 


26.0 


24.2 


27.8 


Highest degree or diploma earned by mother (%) 


Not a high school graduate 


14.4 


14.2 


14.5 


High school diploma or GED 


30.1 


28.1 


32.0 


Occupational or technical certificate or associate's degree 


26.9 


28.1 


25.7 


Bachelor's degree or higher 


11.6 


11.7 


11.5 


Missing 


17.0 


17.8 


16.2 


Working personal computer in home (%) 


72.3 


76.6 


68.0 *** 


Computer has Internet access (%) 


68.6 


72.1 


65.1 ** 


Own or have access to a working car (%) 


82.2 


84.8 


79.5 ** 


Sample size 


911 


458 


453 



SOURCE: MDRC calculations using Baseline Information Form (BIF) data. 

NOTES: A two-tailed t-test was applied to differences between the groups. Statistical significance levels are 
indicated as: * = 10 percent; ** = 5 percent; and *** = 1 percent. 

This table displays most, but not all of the baseline variables that were checked for significance. 

Estimates are adjusted by campus and research cohort. 

Distributions may not add to 100 percent because of rounding. 

“Respondents who said they are Hispanic and chose a race are included only in the Hispanic category. 
Respondents who said they are not Hispanic and chose more than one race are only in the multiracial category. 
’’Distributions may not add to 100 percent because categories are not mutually exclusive. 

in the student success course and at the college overall, as well as information on their grades, 
enrollment status, and credits earned. This report presents information on a number of academic 
outcomes during the semester in which the program was offered as well as three additional 
semesters after the program semester. 

Finally, a survey was disseminated to students in order to collect information on their 
social and emotional perspectives. All students enrolled in the study (both program and control 
group students) were presented with an opportunity to complete the survey during the semester 
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following the program semester. The survey questions were designed to gather data on the 
personal beliefs and habits that the course was expected to affect. Additionally, the survey 
assessed students’ use of and participation in other campus services, such as advising, career 
services, and counseling. 

Qualitative Data 

Researchers also collected qualitative data on Guilford’s success course in two, two-day 
site visits to the college in spring 2008 and fall 2008. Researchers interviewed faculty, staff, and 
administrators in charge of designing and implementing the student success course. These 
interviews yielded important information about the program’s design, the challenges of imple- 
menting the program, the training of the faculty and staff who led the courses, and the contrast 
between the services that students received in the success course and those offered by other 
programs on campus. Additionally, the research staff conducted focus groups with a small 
subset of program group students and students who did not participate in the success course to 
understand more about their experiences in college and, for the program group, their exper- 
iences in the student success course. Researchers also observed several of the success courses 
during both semesters in order to assess whether differences existed among the classes as well 
as in the implementation of the course across different semesters. 



Conclusion 

GuUford was able to identify a clear target group for its success course intervention, which 
would provide a strong contrast between those who received the course (the program group) 
and those who did not (the control group). Though there were some minor differences between 
the program and control group students recmited into the study, the two groups had relatively 
similar characteristics overall, with both groups facing a number of social and financial chal- 
lenges. Researchers’ visits to the college during the spring and fall 2008 semesters allowed for a 
more in-depth investigation of these issues and an opportunity to monitor the development of 
Guilford’s student success course on the ground. The following chapter will discuss what 
researchers observed during these visits and whether the success course was implemented as 
program designers had intended. 
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Chapter 3 

Implementation of the Student Success Course 



This chapter describes the design of Guilford’s student success course for developmental 
education students and its operation during the study period. This chapter draws on information 
from the interviews, focus groups, and classroom observations conducted during the two site 
visits to the college in spring 2008 and fall 2008. It begins with a discussion of the administra- 
tion and start-up of the program and provides an overview of the training provided to the faculty 
and staff teaching the course. It then discusses the implementation of the course and the teach- 
ing and learning provided in the classroom. 

The key findings from this chapter are: 

• The level of training for teaching the success course varied by staff person. 
However, aU faculty and staff teaching the student success course received at 
least the minimum three-day training required for teaching the On Course 
curriculum. 

• While all faculty and staff teaching the course received training, teachers in 
the first semester reported receiving more ongoing support and experiencing 
a greater level of excitement about teaching the curriculum than instructors 
who taught during subsequent semesters. 

• The curriculum and instmction in the success course was relatively standard 
across classrooms, with students receiving similar lessons and assignments 
across different course sections. 



Implementation of the On Course Philosophy in Guilford’s 
Student Success Course 

Though Guilford had offered a more general study skills course to students for several semes- 
ters, a small group of administrators, including the college’s interim vice president of instruc- 
tion, the director of advising, and the director of developmental education, along with several 
faculty and staff, became interested in providing a more intensive opportunity for students to 
explore their personal and career goals and consider the habits that might promote their success 
in college. When the college joined Achieving the Dream in 2004, these leaders used this 
opportunity to explore new strategies for promoting students’ success. In particular, they 
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became interested in exploring new models for revamping their study skills course so that it 
would promote more success-oriented behaviors and habits for their students. ' 

While a number of different published curricula exist for these courses,^ Guilford chose 
to implement a socioemotional-based curriculum, On Course: Strategies for Creating Success 
in College and in Life, developed by Skip Downing.^ This curriculum was designed specifically 
to help students overcome the personal and academic challenges that might impede their 
success in college. Guilford was particularly enthusiastic about the On Course curriculum’s 
focus on “eight core principles” aimed at helping students become more active and responsible 
learners. As can be seen in Box 3.1, these core principles tend to center on modifying students’ 
personal habits, such as heightening their sense of self-responsibility and self-management, or 
improving their interpersonal skills, including increasing students’ awareness of others’ emo- 
tions and perspectives. In order to foster students’ development of these skills, Downing’s 
curriculum employs a number of different active and collaborative teaching strategies, such as 
small student-led working groups, informal class presentations, and intensive journal writing. 
These assignments are offered alongside more traditional course work, such as reading assign- 
ments, essays, quizzes, and a formal class project, which are intended to improve students’ 
academic skills. Guilford administrators and faculty hoped that this dual focus would improve 
both students’ academic achievement and their socioemotional skills. 

Several of these leaders learned about the On Course curriculum through national con- 
ferences, Internet research, and their discussions with administrators at other community 
colleges. After attending introductory sessions at a national training conference, this group of 
leaders began implementing a modified, one-credit study skills course, using the On Course 
curriculum, for students majoring in business technology in fall 2005. Rather than being 
devoted solely to improving study skills, the new course now focused on developing students’ 
personal and work habits, such as promoting students’ time management skills and sense of 
personal responsibility, while also developing their career planning and academic skills. The 
college’s institutional researcher monitored the results of this revised course during the 2005- 
2006 academic year and found promising increases in students’ persistence as well as a bump in 
the three-year graduation rates for students who had taken the student success course. They also 
noted that the course appeared to have the potential to reduce racial achievement gaps for 
African-American male students, as the college observed an increase in the fall-to-spring 
persistence rates for African-American men taking the course."^ 



'Zachry and Qrr (2009). 

^Ellis (2008); Downing (2008); Kanar (2010); Holschuh and Nist-Olejnik (2010). 
^Downing (2008). 

'‘Zachry and Qrr (2009). 
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Box 3.1 


Core Principles of the On Course Curriculum 


Core Principles 


Definition 


Accepting Personal Responsibility 


Seeing oneself as a primary cause of one’s 
outcomes and experiences 


Self-Motivation 


Linding purpose in life by discovering personally 
meaningful goals 


Self-Management 


Consistently planning and taking purposeful action 
in pursuit of goals 


Interdependence 


Building mutually supportive relationships that 
foster the achievement of goals 


Self-Awareness 


Awareness of one’s own beliefs, attitudes, and 
behaviors 


Lifelong Learning 


Linding valuable lessons and wisdom in personal 
experiences 


Emotional Intelligence 


Eifectively managing emotions in support of goals 


Belief in Self 


Seeing oneself as a capable, loveable, and an 
unconditionally worthy human being 


SOURCE; Downing (2008). 





A Success Course for Developmental Education Students 

After noting the promising results for business technology students’ achievement, Guilford 
leaders became interested in offering the revised student success course to other students at the 
college. They particularly focused on providing an expanded version of the course for students 
who came to college with developmental, or remedial, needs in math, English, and reading. 
These leaders also hoped that a two-credit version of this success course would provide even 
more opportunities for students to develop the personal habits and academic skills needed to be 
successful in college. In fall 2007, Guilford implemented a new two-credit pilot of this course 
geared toward students who were taking one or more developmental education classes. This 
pilot course provided a trial mn for the course before the random assignment study of the course 
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began and allowed the leadership team to assess and modify the course pedagogy and curricu- 
lum over the course of the semester. 

The new success course targeted to developmental education students began in earnest 
in the spring 2008 semester, when formal study of the program began. Similar to the fall 2007 
pilot, Guilford offered its new two-credit success course to students who were required to take 
one or more developmental education courses, either in reading, English, or math. A number of 
different sections of the course were offered throughout the day, allowing students several 
choices for enrollment. Though only students who were assessed as having developmental 
needs were qualified to take this course, students were not required to co-enroll in the success 
course and a developmental education or other course. Instead, the success course was offered 
as a stand-alone course. 



Staffing and Training 

Initially, when the student success course was piloted with developmental education students, 
the faculty, staff, and administrators who had originally been trained in the On Course philoso- 
phy were responsible for teaching the course. These instructors received an intensive level of 
training, including a three-day seminar led by Skip Downing on the philosophy behind and 
basic mechanics of the curriculum. Some of these instmctors also participated in a week-long 
seminar, which provided training on the implementation of the On Course curriculum and skills 
needed to train other instmctors in the On Course philosophy. As well as teaching during the 
spring 2008 semester, many of these leaders were also responsible for recmiting more faculty to 
teach the course during subsequent semesters, as well as ensuring that these faculty and staff 
received the training and support they needed to effectively teach the curriculum. In total, 
approximately 30 administrators, faculty, and staff served as instmctors of the success course 
over the three semesters of this study. 

During the three semesters of the study, faculty and staff teaching the course received 
varying levels of professional development. Virtually all instmctors received the basic training 
required to teach the course, which included a three-day seminar introducing new instmctors to 
the On Course pedagogy, course materials, activities, and assignments. Instmctors were either 
trained at Guilford, by a few of the instmctors who had received advanced training, or at Down- 
ing’s introductory institute off campus. Approximately 20 percent of the faculty and staff 
teaching the course also received more advanced preparation, which consisted of an additional 
On Course training with a deeper focus on the course pedagogy and curriculum. Finally, some 
faculty took advantage of other professional development opportunities, such as conferences and 
on-campus workshops, which provided them with ideas and strategies for teaching the course. 
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While the success course instructors received a substantial amount of instructional 
training, interviews and focus groups during spring and fall 2008 revealed that some newer 
instmctors felt that they did not receive enough ongoing support for teaching the course. 
During the first semester of study, the 24 instmctors met monthly to discuss their teaching and 
share best practices. However, in the second semester of the study, these monthly meetings 
were replaced by one meeting at the beginning of the semester to orient the new instructors. In 
lieu of the monthly meetings, newer instmctors were encouraged to set up one-on-one ap- 
pointments with more seasoned instmctors when they felt that they needed additional support. 
However, because many of the new instmctors did not take advantage of the meetings, they 
did not benefit from this help. 

Differences were also noted in instmctors’ level of excitement about the curriculum 
during the second semester of the course. Several new instmctors interviewed in fall 2008 
communicated less enthusiasm about the pedagogy of the course and a limited knowledge of 
the course’s expectations. Several of these individuals also mentioned difficulties in handling 
students’ psychosocial problems, particularly when they involved more serious emotional 
difficulties, such as family problems or suicidal tendencies. Given the depth of students’ 
socioemotional problems and the frequency with which these issues came up in the course, 
several of these new instmctors suggested that only those with a counseling background should 
teach the course. 

There were also noticeable differences in seasoned instmctors’ enthusiasm about the 
course during the second semester of its implementation. In some instances, these instmctors 
were less excited about the course because they had not seen the kind of changes that they were 
expecting in students’ behaviors during their first semester teaching. In focus groups, several of 
these teachers wondered whether the course was helping their students tackle some of their 
personal and academic challenges in the ways that they had originally hoped. Because of these 
differences in their instmctors’ attitudes, students who took the student success course in the 
later implementation period may have had a qualitatively different experience in the classroom 
from that of students who took the course during the first semester of its implementation. 



Participation in the Student Success Course 

In general, the sample of students met Guilford’s eligibility requirements, which indicates that 
the college was successful in recmiting from its target population. As Table 3.1 shows, almost 
90 percent of program and control group students had fewer than 20 credits. Additionally, very 
few program or control group students had any previous exposure to a student success course. 
Finally, relatively few control group students registered for the success course, revealing that 
Guilford was successful in limiting the course to students in the program group. 
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Achieving the Dream: Community Colleges Count 
Table 3.1 



Program Requirements/Participation 
Guilford Technical Community College 



Outcome (%) 


Program 

Group 


Control 

Group 


Difference 


Standard 

Error 


Prior to random assignment 










Eewer than 20 cumulative credits^ 


88.6 


87.3 


1.3 


2.0 


Previously registered for a Study Skills course‘s 


0.0 


1.3 


-1.3 ** 


0.5 


Program semester‘s 










Registered for any courses 


85.8 


84.1 


1.7 


3.7 


Registered for Study Skills course"* 


71.6 


3.1 


68.5 *** 


5.6 


Withdrew from Study Skills course during semester® 


11.1 


0.9 


10.2 *** 


1.8 


Sample size (total = 911) 


458 


453 







SOURCE: MDRC calculations from Guilford Technical Community College transcript data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels 
are indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

““Calculated based on transcripts from summer 2004 to random assignment. 

•“Study Skills courses previously available were AC A 1 1 1 or AC A 118. 

““Estimates are adjusted by campus and research cohort. Standard errors are clustered by Study Skills 
course (AC A 118) section for the program group. 

“*ACA 118: registration was calculated after the add/drop period. 

“=ACA 118: withdrawal was calculated after the add/drop period. 



However, some challenges arose regarding program students’ actual participation in 
the course. A total of 458 students were assigned to the program group during the three 
semesters of the study and registered for one of Guilford’s student success courses. However, 
as can be seen in Table 3.1, nearly 30 percent of these students were no longer registered for 
the success course after the close of the college’s add/drop period, a few days after classes 
began (see Box 3.2 for more information on how to read impact tables). Additionally, another 
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Box 3.2 



How to Read Tables in This Report 

Most tables in this report use a similar format, illustrated below. The abbreviated table below 
displays transeript data and shows some educational outcomes for the program and control 
groups. The first row, for example, shows that program group students attempted an average of 
3.2 courses during the program semester and control group students attempted an average of 
2.9 courses during the program semester. 

Because individuals were assigned randomly either to the program or control group, the effects 
of the program can be estimated by showing the difference in outcomes between the two 
groups. The “Difference” column in the table shows the differences between the two research 
groups’ outcomes — that is, the program’s estimated impacts on the outcomes. For example, 
the average estimated impact on number of courses attempted can be calculated by subtracting 
2.9 from 3.2, yielding an estimated average impact of .03 courses attempted. This difference 
represents the estimated impact of the program rather than the true impact, because although 
study participants are randomly assigned to the program and control groups, differences can 
still be observed by chance. 

Differences marked with one or more asterisks are deemed “statistically significant,” meaning 
that if the true impact is zero, then there is only a small probability of observing, by chance, an 
impact as large (or larger) than the one observed. The number of asterisks indicates the proba- 
bility of observing differences at least as extreme as the observed differences, if the program’s 
true impact is zero. One asterisk corresponds to a 10 percent probability; two asterisks, a 5 
percent probability; and three asterisks, a 1 percent probability. For example, as the first row of 
the table excerpt shows, the program’s estimated impact on number of courses attempted 
during the program semester is 0.3. The one asterisk indicates that this difference is statistical- 
ly significant at the 10 percent level, meaning that there is less than a 10 percent chance of 
observing a difference this large if the program’s true impact is zero. 

The right-hand column of the impact tables in this report provides more information about the 
differences. The standard error of the impact estimate is the measure of uncertainty associated 
with it, and this is used to calculate the statistical significance of the impact. Additionally, 
some tables in this report show effect size as well as standard error. The effect size provides a 
way to interpret the substantive significance of an effect. It is calculated as the difference 
between the outcomes for the program and control group, divided by the standard deviation of 
the control group. Thus the effect size “standardizes” the impacts across measures. 



Outcome 


Program 

Group 


Control 

Group 


Difference 


Standard 

Error 


Number of courses attempted 


3.2 


2.9 


0.3 * 


0.2 


Number of credits attempted 


10.2 


10.0 


0.2 


0.5 


Regular credits 


4.8 


4.5 


0.3 


0.4 


Equated credits 


5.4 


5.5 


-0.1 


0.4 
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11 percent of program students formally withdrew from the success course during the pro- 
gram semester.^ Therefore, only 61 percent of the program group actually received the full 
success course treatment. 

In essence, this drop in participation reflected a normal pattern, as students typically 
shift their course-taking before and during the semester. As with many course registrations, 
weeks or months often ensued between the time of random assignment and registration and the 
first day of classes, giving students ample time to reorganize their schedules and drop the class. 
Furthermore, a number of students withdrew from the course before the end of the add/drop 
period, as is common during the beginning days of a college semester. In addition, Guilford 
drops all students who have not yet paid their tuition for the semester or are not in good finan- 
cial standing at the close of the add/drop period. Guilford did not make an exception to this mle 
for the students in this study, meaning that a number of students may have inadvertently been 
dropped from the course. 

Students in the success course also highlighted another reason for the withdrawal rate. 
In focus groups, a number of students explained that they felt the description of the class in 
Guilford’s course catalogue was misleading, as it described the success course as a study skills 
course.® Upon attending the class, they were surprised to find out that much of the class was 
devoted to personal reflection and journal writing rather than to study habits and academic skill 
development. While these students adjusted to the different course content, they noted that a 
number of others had dropped the course because it did not meet their expectations. 

Though Guilford could increase students’ participation by making the success course 
mandatory, findings from this study do not suggest that increased participation would result in a 
significant change in students’ outcomes (see Chapter 4 for more information). 



Instruction and Learning in the Student Success Course 

GuUford’s student success course was generally taught by two instmctors and stmctured around 
the On Course curriculum, which centered on developing students’ socioemotional skills along 
with their academic abilities. In general, the course employed a more active and collaborative 
pedagogy, with instmctors acting less as lecturers and more as facilitators of group discussions 
and activities. Lessons tended to engage students in critical reflection about their personal 
experiences and habits, in an effort to promote their awareness of their own role in their learning 

® About a third of those students also withdrew from all their other courses. 

®In order to ease the transition from one college to another across colleges in the state. North Carolina has 
state-mandated descriptions for most community college course offerings. As a result, Guilford’s course 
description for the student success course still noted a focus on study skills, even though the curriculum for the 
course had been markedly changed. 
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and future responsibilities. As seen in Box 3.3, students were encouraged to reflect upon their 
role in their learning through assignments such as weekly journal writing or sharing their 
personal experiences with the class. These activities were expected to increase students’ 
understanding of and commitment to their own as well as others’ learning, thus providing them 
with important skills for further defining their college and career choices. 

Students also received some instmction and practice in study and academic skills. For 
instance, some of the course lessons focused on note-taking, time management, and test 
preparation. In addition to journal writing, students were also required to complete a more 
formal term paper and class presentation, which helped them hone some of their writing and 
grammar skills. In general, success course instructors actively engaged students in this learning 
process through activities such as group readings, projects, and classroom discussions; however, 
at times, teachers took a more “stand and deliver’’ approach when reviewing for quizzes or tests. 
A consistent thread throughout most sections of the course was that instmctors made an effort to 
boost students’ confidence and self-esteem. 



Contrast Between Program Group Students’ and Control Group 
Students’ Experiences 

Socioemotional and academic skill-building was a distinct hallmark of Guilford’s success 
course and was intended to provide a unique support to students enrolled in the course. As noted 
earlier, students assigned to the program group were expected to take the course while control 
group students were barred from taking it. However, both program and control group students 
had access to the college’s other support programs, including services such as advising, tutoring 
in academic labs, and career counseling (see Box 3.3). 

Given that some of Guilford’s developmental courses also focused on improving stu- 
dents’ study and academic skills and that some crossover existed with instmctors in these two 
areas, some concerns were raised about the uniqueness of Guilford’s success course curriculum 
and focus. For instance, interviews revealed that some developmental education instructors 
believed that they regularly did “hand-holding’’ with their students, teaching them skills such as 
how to keep a notebook and the importance of being on time. However, while these develop- 
mental faculty at times covered topics similar to those in the success course curriculum, they 
emphasized that the content and approach of the student success course differed from that of 
their developmental courses. For example, faculty in developmental English courses tended to 
focus primarily on teaching reading and writing rather than having students share their personal 
experiences. In contrast, the student success instmctors tended to take on a “life coach’’ or 
counselor role, modeling particular habits and behaviors for students. Similarly, the success 
course focused primarily on students’ personal problems and soft skills while developmental 
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Box 3.3 



Differences in Services Provided to Program and 
Controi Group Students 



Program Group Experience 


Control Group Experience 


Access to support services offered to all 
students, such as: 


Access to support services offered to all 
students, such as: 


• Advising 


• Advising 


• Tutoring in the math and writing 
lahs 


• Tutoring in the math and writing 
labs 


• Career counseling in career center 


• Career counseling in career center 


• Information on transferring to a four- 
year college at the College Transfer 
Center 


• Information on transferring to a 
four-year college at the College 
Transfer Center 


Required enrollment in a two-credit, 
semester-long student success course 


Barred from enrolling in the student 
success course 


• Training in and reflection on 

socioemotional aspects of learning, 
such as self-management, self- 
responsihility, and awareness of per- 
sonal learning styles 




• Collaborative learning projects and 
discussions intended to increase 
interpersonal skills and emotional 
intelligence 




• Instruction and practice in selected 
academic skills, such as writing and 
grammar 




• Training in study skills, such as time 
management, note taking, and test 
preparation 




• Development of public speaking 
skills through formal and informal 
classroom presentations 




• Additional support and encourage- 
ment to use needed student services 
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courses were devoted to academic instmction. While there was a minor focus on soft skills in 
developmental courses, instructors tended to address these issues in passing and rarely had time 
to fully address these matters in their academic courses. 

Students also emphasized the marked differences in the activities and assignments they 
undertook in their success course in contrast to their developmental education or other academic 
courses. In focus groups with researchers, most students said that they felt a real connection 
with their student success instructors and believed that these instmctors cared about their 
success, which was not always the case with their academic instmctors. Additionally, students 
noted teachers’ different approach to the success course material in contrast with their academic 
courses, even when they had the same teacher for both. 



Conclusion 

In sum, while Guilford’s success course for developmental education students aligned with the 
On Course curriculum and ideals, a number of challenges also arose in the implementation of 
the course over the course of the three semesters. First, instmctors’ enthusiasm for the course 
and support for its implementation waned somewhat over the final two semesters of the study, 
which may have led some students to have a different course experience. Additionally, a 
substantial proportion of students who had originally enrolled in the class withdrew by the end 
of the program semester, so that only a subset of students received the full success course 
treatment. These issues may have led a number of students in the program group to receive 
qualitatively different experiences of the course than the program designers had intended and 
may have reduced their exposure to the course’s ideology. 
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Chapter 4 

Impacts of the Student Success Course 



This chapter highlights the impacts of Guilford’s student success course on students’ socioemo- 
tional skills and well-being as well as their academic achievement. The chapter begins with a 
discussion of the findings from a survey of both program and control group students, which 
asked about their changes in personal habits and use of campus services. The analysis then 
moves on to the effects of the course on students’ academic performance, including their 
persistence, grades, courses attempted, and credits earned. The chapter concludes with a 
discussion of the differential findings from a subgroup of students who took the success course 
during the first semester of its implementation. 

The key findings from this chapter are: 

• Guilford’s student success course had a positive impact on students’ self- 
management, interdependence, self-awareness, interest in lifelong learning, 
emotional intelligence, and positive engagement in college, among students 
who had low levels of these attributes. The program had no effect on stu- 
dents’ acceptance of personal responsibility, self-motivation, and use of Guil- 
ford’s support services. 

• Though Guilford’s success course had some positive effects on students’ so- 
cial and psychological well-being, it had no meaningful effects on students’ 
academic achievement, including course registration or enrollment, credits 
attempted, course pass rates, course withdrawal rates, credits earned, success- 
ful completion of developmental education courses, or grade point average. 

This lack of meaningful academic impacts suggests that the social and psy- 
chological impacts may not have been strong enough to affect students’ 
achievement and that other programs and services may be needed to improve 
their academic performance. 

• Guilford’s student success course had a differential effect on the first cohort 
of students who enrolled in the study. However, it is unclear whether this ef- 
fect was a result of different experiences in the course by the first cohort of 
students, differences in the control groups’ performance across the cohorts, 
or other factors. 
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Effects on Students’ Socioemotional Skills and Well-Being 

Guilford administrators hoped that the student success course would have a positive impact on 
students’ self-esteem, life skills, and attachment to the college. In order to assess whether the 
course made a difference in these socioemotional characteristics, MDRC developed a survey 
that was administered to both the program and the control group and asked students to reflect on 
any changes in their personal habits and self-awareness over the previous semester. It focused 
primarily on aspects of the On Course curriculum’s eight core principles, including self- 
management, self-responsibility, self-motivation, interdependence, self-awareness, belief in self, 
emotional intelligence, and interest in lifelong learning. Many of the questions were revised 
statements taken from the curriculum developer’s own questionnaires. However, several 
questions were also asked about students’ engagement in college and their use of particular 
college services, which were drawn from the Community College Survey of Student Engage- 
ment or from other MDRC surveys. ' 

The survey was administered to students within six months of the program semester. 
Students were presented with a series of statements and asked to indicate how much they agreed 
or disagreed with each statement on a five-item scale that ranged from “strongly disagree’’ to 
“strongly agree.’’ Additionally, they were asked to rate how often they used various student 
services on a three-item scale ranging from “rarely/never’’ to “often’’ and how often they 
experienced different indicators of positive engagement on a four-item scale ranging from 
“never” to “very often.” See Appendix A for more details on the survey scales. Just over 70 
percent of participants responded to the survey. Although a higher response rate is usually 
preferable, few differences existed in the baseline characteristics of the program and control 
group students who responded,^ making it reasonable to assume that the impacts in survey 
outcomes were caused by the program itself. 

The survey results are displayed in Table 4.1, which details students’ self-assessment of 
their own psychosocial competencies and skills, their use of Guilford’s services, and their 
engagement in college. To facilitate interpretation of the students’ self-assessment, the table 
presents the proportion of students that had high levels and low levels of these characteristics. 



*The Community College Survey of Student Engagement, administered annually at community colleges 
across the country, asks students about their college experiences in order to assess institutional practices and 
student behaviors that are highly correlated with student learning and student retention. Community College 
Survey of Student Engagement (201 1). 

^Separate tests for differences in baseline characteristics were performed on the survey sample. The results 
were similar to those of the overall sample, with slightly fewer significant differences appearing in the survey 
sample. 
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Achieving the Dream: Community Colleges Count 
Table 4.1 



Survey Outcomes 

Guilford Technical Community College 



Outcome (%) 


Program 

Group 


Control 

Group 


Difference 


Standard 

Error 


Effect 

Size 


Accepting personal responsibility'* 


Low 


14.2 


12.4 


1.9 


2.4 


0.1 


High 


26.9 


25.5 


1.4 


2.9 


0.0 


Self-motivation'* 


Low 


20.0 


22.4 


-2.4 


2.8 


-0.1 


High 


0.0 


0.0 


0.0 


0.0 


0.0 


Self-management" 


Low 


13.9 


20.0 


-6.1 ** 


2.6 


-0.2 


High 


27.1 


23.4 


3.7 


3.3 


0.1 


Interdependence"* 


Low 


10.5 


18.2 


_7 ^ *** 


2.8 


-0.2 


High 


22.0 


17.0 


5.0 


3.3 


0.1 


Self-awareness" 


Low 


12.7 


18.2 


-5.5 ** 


2.7 


-0.1 


High 


27.0 


23.8 


3.2 


3.0 


0.1 


Lifelong learning^ 


Low 


8.4 


14.0 


-5.6 ** 


2.4 


-0.2 


High 


24.6 


24.1 


0.5 


3.4 


0.0 


Emotional intelligence® 


Low 


7.5 


16.4 


_g 9 *** 


3.0 


-0.2 


High 


18.8 


19.6 


-0.9 


2.8 


0.0 


Belief in self** 


Low 


7.8 


14.0 


-6.2 ** 


2.7 


-0.2 


High 


26.6 


25.4 


1.2 


3.1 


0.0 


Use of GTCC services' 


Low 


11.6 


15.5 


-3.9 


3.0 


-0.1 


High 


16.5 


15.7 


0.8 


3.4 


0.0 


Positive engagement ^ 


Low 


10.8 


18.0 


_7 2 *** 


2.7 


-0.2 


High 


14.3 


15.7 


-1.4 


2.5 


0.0 


Sample size (total = 661 ) 


329 


332 









(continued) 
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Table 4.1 (continued) 



SOURCE: MDRC calculations from Guilford Technical Community College survey data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; = 5 percent; * = 10 percent. 

Estimates are adjusted hy campus and research cohort. 

Standard errors are clustered hy the Study Skills course (AC A 118) section for the program group. 

The "self- management," "interdependence," "self-awareness," "lifelong learning," and "positive engagement" 
scales had actual positive impacts for the overall scale as well as the low part of the distrihution's impact indicated 
by the stars in the table. The "belief in self" scale's impact shown in the table does not translate to a positive 
impact for the overall scale as the other impacts do. 

"Low" is the percentage of sample members scoring one standard deviation below the mean; "high" is the 
percentage of sample members scoring one standard deviation above the mean. 

“5-item scale about feelings of personal responsibility; response categories range from 1 = "strongly disagree" 
to 5 = "strongly agree." The five items are averaged. 

’’3-item scale about feelings of self-motivation; response categories range from 1 = "strongly disagree" to 5 = 
"strongly agree." The three items are averaged. 

‘’4-item scale about feelings of self-management; response categories range from 1 = "strongly disagree" to 5 = 
"strongly agree." The four items are averaged. 

‘’5-item scale about feelings of interdependence; response categories range from 1 = "strongly disagree" to 5 = 
"strongly agree." The four items are averaged. 

“T-item scale about feelings of self-awareness; response categories range from 1 = "strongly disagree" to 5 = 
"strongly agree." The five items are averaged. 

’4-item scale about feelings of lifelong learning; response categories range from 1 = "strongly disagree" to 5 = 
"strongly agree." The four items are averaged. 

64-item scale about feelings of emotional intelligence; response categories range from 1 = "strongly disagree" 
to 5 = "strongly agree." The four items are averaged. 

’’5-item scale about feelings of belief in self; response categories range from 1 = "strongly disagree" to 5 = 
"strongly agree." The five items are averaged. 

‘9-item scale about frequency of use of GTCC services; response categories range from 1 = "rarely/never" to 3 
= "often." The nine items are averaged. 

j 17-item scale about frequencies of positive student engagement; response categories range from 1 = "never" to 
4 = "very often." The seventeen items are averaged. 



respectively.^ As can be seen in the table, a larger proportion of control group students fall into 
the lower levels of self-management, interdependence, self-awareness, interest in lifelong 
learning, emotional intelligence, and positive engagement in college than do program group 
students."^ However, there are no significant differences in the proportion of control and program 
group students exhibiting higher levels of these characteristics. This indicates that the program 
changed the distribution of these scales more by moving students out of the lowest levels than 
by moving students to the highest levels. In other words, the student success course appears to 

^Students scoring in the high range were 1 standard deviation above the average score; students scoring in 
the low range were 1 standard deviation below the average. 

"^These measures had positive impacts for the overall scale as well as the low part of the distribution’s im- 
pact indicated by stars in Table 4.1. The “belief in self’ scale’s impact shown on Table 4.1 does not translate to 
a positive impact for the overall scale as the other impacts do. 
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have benefitted students with lower levels of self-management, interdependence, self- 
awareness, interest in lifelong learning, emotional intelligence and engagement in college, 
helping them to develop more of these skills. The success course had no impact on a few 
characteristics, including students’ acceptance of personal responsibility, self-motivation, and 
use of Guilford’s support services. See Table 4.1 and the description of survey scales in Appen- 
dix A for more details. 

Therefore, it appears that Guilford’s success course was able to help students with low- 
er psychosocial skills and abilities to become more aware of themselves and some of the ways 
that they could affect their own educational and personal trajectories. These results are hearten- 
ing, as this was the subset of students that the faculty and staff who implemented Guilford’s 
revised success course were particularly interested in reaching. Unfortunately, the course did 
not appear to have an impact on these students’ awareness of themselves as the primary cause 
of their outcomes and experiences (self-responsibility), their discovery of personally meaningful 
goals (self-motivation), or their use of services such as tutoring and advising. This suggests that 
while the course was able influence some aspects of students’ attitudes and behaviors, other 
interventions may be needed to influence their sense of self-responsibility, self-motivation, and 
their use of student support services. 



Academic Findings 

In addition to influencing students’ social and emotional habits and behaviors, Guilford’s 
student success course curriculum was designed to help students develop better study and 
academic skills. The course’s designers hoped that improvements in students’ study and 
personal habits might also lead to higher levels of academic achievement, including success in 
their academic courses, higher persistence rates in college, and better progress through devel- 
opmental education. Using transcript data, this section describes the effects of the student 
success course on various educational outcomes during sample members’ first semester in the 
study (the “program semester”) and three postprogram semesters. 

Effects on Educational Outcomes During the Program Semester 

As can be seen in Table 4.2, few significant differences existed between the academic 
outcomes of program and control group students during the semester in which the program 
operated (see Box 3.2 for more information on how to read these tables). Initially, a significant 
difference was detected in the number of courses program and control group students attempted 
in the program semester, with program group students taking about a th ir d of a course more 
than control group students, on average. However, this difference is driven by the student 
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Achieving the Dream: Community Colleges Count 
Table 4.2 



Academic Outcomes in Program Semester 
Guilford Technical Community College 



Outcome 


Program 

Group 


Control 

Group 


Difference 


Standard 

Error 


Registered for any courses (%)'* 


85.8 


84.1 


1.7 


3.7 


Registered for study skills course (%) 


71.6 


3.1 


68.5 *** 


5.6 


Number of courses attempted 


3.2 


2.9 


0.3 * 


0.2 


Number of credits attempted 


10.2 


10.0 


0.2 


0.5 


Regular credits 


4.8 


4.5 


0.3 


0.4 


Equated credits 


5.4 


5.5 


-0.1 


0.4 


Attempted Developmental Math (%) 


62.1 


61.3 


0.8 


3.6 


Attempted Developmental English (%) 


37.6 


39.5 


-1.9 


4.2 


Attempted Developmental Reading (%) 


37.7 


38.2 


-0.5 


3.7 


Passed all courses (%) 


31.7 


35.7 


-3.9 


3.0 


Withdrew from any courses (%) 


35.5 


35.7 


-0.2 


3.7 


Number of credits earned 


6.1 


6.4 


-0.3 


0.4 


Regular credits 


3.0 


2.9 


0.1 


0.3 


Equated credits 


3.1 


3.6 


-0.4 


0.3 


Passed Developmental Math (%) 


34.9 


37.6 


-2.7 


3.5 


Passed Developmental English (%) 


21.5 


26.6 


-5.1 


3.1 


Passed Developmental Reading (%) 


22.8 


25.9 


-3.1 


3.1 


Term GPA (%) 










No GPA’’ 


54.7 


49.7 


5.0 


3.9 


3.0 to 4.0 or B/A 


21.1 


21.9 


-0.8 


2.7 


2.0 to 2.9 or C/B- 


11.0 


12.3 


-1.3 


2.1 


1.0 to 1.9orD/C- 


4.8 


5.7 


-0.9 


1.4 


0 to 0.9 or F/D- 


8.3 


10.3 


-2.0 


1.8 


Sample size (total = 911) 


458 


453 







SOURCE: MDRC calculations from Guilford Technical Community College transcript data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by campus and research cohort. 

Standard errors are clustered by Study Skills course (ACA 118) section for the program group. 

‘‘Students who dropped all courses before the drop date are not counted as registered. 

'’The "No GPA" category includes students who were not registered, students who withdrew from all classes, 
and students who took only developmental courses, which are not included in GPA calculations. The Study Skills 
course (ACA 118) was not included in GPA calculations for this table. 
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success course itself. When excluding the success course, the differences in courses attempted 
are reversed,^ with control group students taking about a third of a course more than the pro- 
gram group. 

This reveals that while the success course may have added to the number of courses 
students attempted overall, it actually lowered the number of other credit or developmental 
courses students took during the program semester. However, given that Guilford’s success 
course was offered as a two-credit course, this finding is not surprising, as the success course 
might have crowded out other credit courses that program students could have taken. Therefore, 
the more important question is whether the student success course was able to have an impact 
on students’ performance in their other courses, thus demonstrating the utility of taking this 
course in place of other courses students might attempt. 

Unfortunately, the student success course did not produce statistically significant ef- 
fects on students’ academic outcomes during the program semester. As can be seen in Table 
4.2, all students, regardless of whether they were program or control group students, had 
academic difficulties, with relatively low course pass rates and credits earned. However, no 
significant differences exist between program and control group students’ academic outcomes, 
including course registration or enrollment, credits attempted, course pass rates, course 
withdrawal rates, credits earned, successful completion of developmental education courses, or 
grade point averages.® This means that the success course did not have a significant impact on 
program group students’ overall achievement during the semester that they took the student 
success course. 

Effects on Educational Outcomes After the Program Semester 

When analyzing the effects of Guilford’s student success course, MDRC examined Guilford 
transcript data through the fall 2010 semester, or up to three semesters after students were 
originally eligible for the program. The analysis below captures the effects of Guilford’s student 
success course though the third semester after students were randomly assigned to the program. 

As with the findings during the program semester, there are no significant differences in 
academic outcomes between the program and control groups in the first postprogram semester 
(see Table 4.3). While enrollment decreased in general during the first postprogram semester 
and students continued to experience academic difficulties, the program and control groups 
were similar. As during the program semester, the student success course did not produce any 



®This alternate version of courses attempted, excluding the success course, is not shown in the tables. 

®The grade point averages (GPA) shown in the tables were calculated excluding the student success course 
in order to examine what impact the student success course had on students’ GPA in other courses. 
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Achieving the Dream: Community Colleges Count 
Table 4.3 



Academic Outcomes in First Postprogram Semester 
Guilford Technical Community College 



Outcome 


Program 

Group 


Control 

Group 


Difference 


Standard 

Error 


First Dosturosram semester 


Registered for any courses {%f 


63.6 


61.6 


2.0 


3.5 


Number of courses attempted 


2.3 


2.2 


0.1 


0.2 


Number of credits attempted 


7.6 


7.3 


0.3 


0.5 


Regular credits 


5.2 


4.9 


0.3 


0.4 


Equated credits 


2.3 


2.4 


0.0 


0.2 


Attempted Developmental Math (%) 


30.2 


27.3 


2.9 


3.1 


Attempted Developmental English (%) 


16.1 


18.4 


-2.4 


2.5 


Attempted Developmental Reading (%) 


12.7 


13.6 


-0.9 


2.4 


Number of credits earned 


4.4 


4.3 


0.1 


0.4 


Regular credits 


3.2 


3.2 


0.0 


0.3 


Equated credits 


1.2 


1.2 


0.1 


0.1 


Passed Developmental Math (%) 


15.4 


13.1 


2.3 


2.2 


Passed Developmental English (%) 


8.7 


8.9 


-0.2 


1.6 


Passed Developmental Reading (%) 


7.1 


7.4 


-0.2 


1.7 


Term GPA (%) 


No GPA 


51.2 


51.8 


-0.6 


3.2 


3.0 to 4.0 or B/A 


19.6 


21.5 


-1.9 


2.7 


2.0 to 2.9 or C/B- 


10.1 


12.1 


-2.1 


2.0 


1.0 to 1.9 orD/C- 


7.2 


4.8 


2.4 


1.7 


0 to 0.9 or F/D- 


12.0 


9.7 


2.3 


1.8 


Sample size (total = 911) 


458 


453 







SOURCE: MDRC calculations from Guilford Technical Community College transcript data. 



NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by campus and research cohort. 

Standard errors are clustered by the Study Skills course (ACA 118) section for the program group 
‘‘Students who dropped all courses before the drop date are not counted as registered. 

'’The "No GPA" category includes students who were not registered, students who withdrew from all 
classes, and students who took only developmental courses, which are not included in GPA calculations. 
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meaningful changes in students’ persistence (as evidenced in students’ registration rates), credits 
or courses attempted, credits earned, successful completion of developmental education courses, 
or grade point averages. 

Extended follow-up of Guilford’s success course demonstrates that there was not any 
change in overall impacts in the longer term, through the third postprogram semester. Table 4.4 
shows that none of the cumulative outcomes in the third postprogram semester is statistically 
different between program and control groups. As was the case in the program semester and the 
first postprogram semester, the student success course did not produce any meaningful changes 
in students’ persistence, credits or courses attempted, credits earned, successful completion of 
developmental education courses, or grade point averages measured cumulatively through four 
semesters beginning with the program semester. 

These findings indicate that while Guilford’s success course was able to change some 
aspects of students’ personal habits, it did not have a meaningful impact on the overall academic 
achievement of these students during the program semester or the semesters following the 
program. However, the success course was found to have some promising effects on certain 
student subgroups, as can be seen in the exploratory analysis discussed below. 



Subgroup Analysis: Effects on Students Enrolled in Courses 
During the Program Semester 

As discussed earlier in Chapter 3, a substantial proportion of students assigned to the program 
group did not actually receive the full success course, either because they did not take any 
classes at Guilford that semester or because they dropped or withdrew from the success course 
during the program semester. It is expected that students’ nonparticipation would affect the 
overall impact of the success course on students’ outcomes. When program members do not 
participate, the differences between the program and control groups are diluted, as the program 
group contains a number of students who did not receive the full treatment. Unfortunately, 
removing nonparticipants Irom the program group would detrimentally affect the integrity of 
the study, as it would allow students to select themselves into the program by their own actions, 
not by random assignment. However, the effect of nonparticipation can be explored through an 
analysis that examines only those students, among both the program and control group, who 
took courses during the program semester.’ 

This exploratory analysis is detailed in Appendix B. It suggests that program students’ 
participation in the student success course may have effects on some academic outcomes. 

’Because dropping students from the sample fundamentally changes the random assignment procedure, 
this analysis should be considered as only exploratory. 
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Achieving the Dream: Community Colleges Count 
Table 4.4 



Academic Outcomes: 

Cumulative from Program to Third Postprogram Semester 
Guilford Technical Community College 



Outcome 


Program 

Group 


Control 

Group 


Difference 


Standard 

Error 


Registered for any courses (%)'“ 


90.6 


90.3 


0.3 


2.7 


Number of semesters registered 


2.4 


2.4 


0.0 


0.1 


Number of credits attempted 


30.2 


29.9 


0.3 


1.6 


Regular credits 


20.1 


19.6 


0.4 


1.4 


Equated credits 


10.1 


10.3 


-0.1 


0.7 


Number of credits earned 


18.4 


18.6 


-0.2 


1.3 


Regular credits 


12.8 


12.7 


0.1 


1.1 


Equated credits 


5.6 


5.9 


-0.3 


0.5 


Passed Developmental Math (%) 


48.8 


47.8 


1.1 


3.2 


Passed Developmental English (%) 


31.4 


32.9 


-1.5 


3.5 


Passed Developmental Reading (%) 


29.2 


30.5 


-1.4 


3.4 


Term GPA (%) 


No GPA 


28.9 


24.7 


4.2 


3.4 


3.0 to 4.0 or B/A 


20.5 


20.8 


-0.3 


2.7 


2.0 to 2.9 or C/B- 


19.2 


22.3 


-3.1 


2.8 


1.0 to 1.9orD/C- 


14.4 


14.6 


-0.2 


2.4 


0 to 0.9 or F/D- 


17.0 


17.7 


-0.7 


2.6 


Sample size (total = 911) 


458 


453 







SOURCE: MDRC calculations from Guilford Technical Community College transcript data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Standard errors are clustered by Study Skills course (ACA 118) section for the program group. 

“Students who dropped all courses before the drop date are not counted as registered. 

'’The "No GPA" category includes students who withdrew from all classes and students who took only 
developmental courses, which are not included in GPA calculations. The Study Skills course (ACA 118) was not 
included in GPA calculations for this table. 
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However, these effects are not consistent across the two semesters. As a result, it is un- 
clear whether a higher participation rate would have made the program more successful in 
improving academic outcomes. 



Subgroup Analysis: Effects on Students by Cohort 

While there were no overall impacts on students’ achievement, an analysis of the effects of the 
student success course on individual student cohorts tells a more promising story. As discussed 
in Chapter 3, the initial cohort of instmctors for the success course was more enthusiastic and 
had more supports than those who taught the course during the second and third semesters. 
Additionally, several new instmctors who taught students in the second and third cohort noted 
challenges in helping students with more severe behavioral and emotional problems. Such 
issues raised questions about whether differences might exist in students’ outcomes among 
these three semesters. 

Table 4.5 and Table 4.6 depict the academic outcomes of the first cohort of students and 
the remaining two cohorts of students during the program semester and the first postprogram 
semester, respectively. As can be seen in these tables, the student success course had a differen- 
tial effect on the first cohort of students that enrolled in the study.® During the program semes- 
ter, the success course reduced the number of students receiving a D- or F in their courses 
among those students in the first cohort (see Table 4.5). In addition, the success course resulted 
in a number of impacts on the first cohort of students’ achievement during the first postprogram 
semester (see Table 4.6). 

Extended follow-up of students by cohort, shown in Table 4.7, reveals that the positive, 
statistically significant impact on regular credits earned for the first cohort is sustained in the 
second and th ir d postprogram semesters, with a marginally increasing cumulative impact on 
regular credits earned at the end of each semester. By the end of the second postprogram 
semester, program group students in the first cohort are 2.3 regular credits (not shown in table) 
ahead of their control group counterparts, and by the end of the th ir d postprogram semester, 
program group students are 3.1 regular credits ahead. As can be seen in Tables 4.5 to 4.7, this 
impact on regular credits earned for the first cohort is driven by higher rates of regular courses 
attempted on average by the program group students. 

Some of this variation in student outcomes may have resulted from the differences in 
the implementation of the success course during these three semesters. As noted in Chapter 3, 



^Because random assignment was performed by cohort, these are unbiased, fully experimental impacts 
within each cohort. Moreover, baseline characteristics do not differ significantly between the spring 2008 
cohorts and the following cohorts. 
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Achieving the Dream: Community Colleges Count 
Table 4.5 



Transcript Outcomes, Program Semester, by Cohort 
Guilford Technical Community College 







Snrine 2008 Cohort 




1 Eall 2008 and Snrine 2009 Cohorts 1 


Difference 




Program Control 




Standard 


Program Control 




Standard 


Between 


Outcome 


Group 


Group Difference 


Error 


Group 


Group Difference 


Error 


Subgroups 


Proeram Semester 




















Registered for any courses (%)“ 


88.5 


86.1 


2.5 


4.9 


84.7 


83.3 


1.4 


4.2 




Registered for study skills course 


71.0 


2.2 


68.8 *** 


7.7 


71.9 


3.4 


68.4 *** 


5.7 




Number of credits attempted 


10.5 


10.1 


0.4 


0.7 


10.0 


9.9 


0.1 


0.6 




Regular credits 


4.9 


4.9 


0.0 


0.6 


4.7 


4.3 


0.4 


0.4 




Equated credits 


5.7 


5.2 


0.4 


0.6 


5.3 


5.6 


-0.3 


0.4 




Attempted Developmental Math (%) 


66.3 


61.9 


4.4 


6.0 


60.3 


61.2 


-0.9 


4.2 




Attempted Developmental English (%) 


38.8 


37.6 


1.2 


7.1 


37.0 


40.4 


-3.4 


4.7 




Attempted Developmental Reading (%) 


37.1 


31.9 


5.3 


6.1 


38.0 


41.0 


-3.0 


4.1 




Number of credits earned 


6.4 


5.9 


0.5 


0.6 


6.0 


6.7 


-0.6 


0.5 




Regular credits 


3.4 


2.7 


0.7 


0.5 


2.8 


3.0 


-0.1 


0.3 




Equated credits 


3.0 


3.2 


-0.2 


0.5 


3.2 


3.7 


-0.5 


0.4 




Passed Developmental Math (%) 


35.2 


33.8 


1.4 


5.9 


34.8 


39.2 


- 4.3 


4.5 




Passed Developmental English (%) 


18.3 


25.0 


-6.7 


4.9 


22.9 


27.2 


-4.3 


3.9 




Passed Developmental Reading (%) 


22.0 


21.4 


0.6 


4.8 


23.2 


27.8 


- 4.6 


3.8 




Term GPA (%) 




















No GPa'’ 


48.6 


44.4 


4.2 


7.2 


57.2 


52.0 


5.2 


4.4 




3.0 to 4.0 or B/A 


21.4 


22.8 


-1.4 


5.6 


21.1 


21.5 


-0.4 


3.7 




2.0 to 2.9 or C/B- 


16.2 


12.3 


3.9 


5.1 


8.8 


12.3 


-3.5 


2.3 




1.0 to 1.9 orD/C- 


6.9 


5.1 


1.8 


2.6 


4.0 


6.0 


-2.0 


1.7 




0 to 0.9 or E/D- 


6.9 


15.4 


-8.5 ** 


3.5 


8.9 


8.2 


0.7 


2.0 


tt 


Sample size (total = 911) 


131 


136 






327 


317 







(continued) 




Table 4.5 (continued) 

SOURCE: MDRC calculations from Guilford Technical Community College transcript data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 
percent; * = 10 percent. 

A two-tailed t-test was applied to differences of impacts between subgroups. Statistical significance levels are indicated as: tt| = 1 percent; ft = 5 
percent; | = 10 percent. 

Estimates are adjusted by campus and research cohort. 

Standard errors are clustered by the Study Skills Course (ACA 118) section for the program group. 

^Students who dropped all courses before the drop date are not counted as registered. 

’’The "No GPA" category includes students who were not registered, students who withdrew from all classes, and students who took only 
developmental courses, which are not included in GPA calculations.The Study Skills course (ACA 118) was not included in GPA calculations for this 
table. 
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Table 4.6 



Transcript Outcomes, First Postprogram Semester, by Cohort 
Guilford Technical Community College 







Spring 2008 Cohort 




1 Fall 2008 and Spring 2009 Cohorts I 


Difference 




Program Control 




Standard 


Program Control 




Standard 


Between 


Outcome 


Group 


Group Difference 


Error 


Group 


Group Difference 


Error 


Subgroups 


First DOStDrosram semester 




















Registered for any courses (%)““ 


63.3 


52.3 


11.0 * 


6.1 


63.7 


65.5 


-1.8 


4.1 


t 


Number of credits attempted 


7.7 


6.1 


1.5 * 


0.8 


7.6 


7.7 


-0.2 


0.6 


t 


Regular credits 


5.9 


4.2 


j 7 ** 


0.7 


5.0 


5.2 


-0.2 


0.5 


tt 


Equated credits 


1.7 


1.9 


-0.2 


0.4 


2.6 


2.5 


0.1 


0.3 




Attempted Developmental Math (%) 


26.5 


20.0 


6.5 


5.1 


31.8 


30.3 


1.4 


4.0 




Attempted Developmental English (%) 


12.2 


16.9 


-4.7 


4.3 


17.6 


19.0 


-1.4 


3.1 




Attempted Developmental Reading (%) 


5.2 


11.9 


-6.8 * 


3.4 


15.8 


14.3 


1.5 


2.8 


T 


Number of credits earned 


4.5 


3.7 


0.8 


0.5 


4.4 


4.6 


-0.2 


0.5 




Regular credits 


3.5 


2.6 


1.0 ** 


0.5 


3.1 


3.4 


-0.3 


0.4 


tt 


Equated credits 


1.0 


1.1 


-0.1 


0.3 


1.3 


1.2 


0.2 


0.2 




Passed Developmental Math (%) 


14.5 


11.1 


3.4 


3.7 


15.8 


13.9 


1.9 


2.7 




Passed Developmental English (%) 


7.0 


10.2 


-3.2 


2.8 


9.4 


8.3 


1.1 


2.0 




Passed Developmental Reading (%) 


3.7 


7.5 


-3.8 


2.6 


8.5 


7.3 


1.3 


2.0 




Term GPA (%) 




















No GPA'’ 


51.0 


58.2 


-7.2 


5.5 


51.2 


49.1 


2.1 


4.0 




3.0 to 4.0 orB/A 


18.3 


18.4 


-0.1 


4.6 


20.1 


22.8 


-2.7 


3.2 




2.0 to 2.9 or C/B- 


10.8 


11.7 


-0.9 


3.6 


9.8 


12.3 


-2.6 


2.3 




1.0 to 1.9 orD/C- 


10.7 


2.2 


8.6 *** 


3.3 


5.8 


6.0 


-0.2 


2.0 


tt 


0 to 0.9 or F/D- 


9.2 


9.6 


-0.4 


4.0 


13.1 


9.8 


3.4 


2.2 




Sample size (total = 911) 


131 


136 






327 


317 
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Table 4.6 (continued) 

SOURCE: MDRC calculations from Guilford Technical Community College transcript data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 
percent; * = 10 percent. 

A two-tailed t-test was applied to differences of impacts between subgroups. Statistical significance levels are indicated as: tt| = 1 percent; ft = 5 
percent; | = 10 percent. 

Estimates are adjusted by campus and research cohort. 

Standard errors are clustered by the Study Skills Course (ACA 118) section for the program group. 

^Students who dropped all courses before the drop date are not counted as registered. 

’’The "No GPA" category includes students who were not registered, students who withdrew from all classes, and students who took only 
developmental courses, which are not included in GPA. 




Achieving the Dream: Community Colleges Count 
Table 4,7 



Cumulative Academic Outcomes, Program to Third Postprogram Semester, by Cohort 

Guilford Technical Community College 







Spring 2008 Cohort 




Fall 2008 and Spring 2009 Cohorts 


Difference 




Program Control 




Standard 


Program Control 




Standard 


Between 


Outcome 


Group 


Group Difference 


Error 


Group 


Group 


Difference 


Error 


Subgroups 


Registered for any courses (%)“ 


93.8 


90.5 


3.3 


3.7 


89.3 


90.2 


-0.9 


3.3 




Number of semesters registered 


2.5 


2.2 


0.2 


0.2 


2.4 


2.5 


-0.1 


0.1 




Number of credits attempted 


31.3 


27.9 


3.4 


2.3 


29.8 


30.7 


-1.0 


1.9 




Regular credits 


21.9 


18.8 


3.2 


2.2 


19.3 


20.0 


-0.6 


1.6 




Equated credits 


9.4 


9.1 


0.3 


0.9 


10.4 


10.8 


-0.3 


0.9 




Number of credits earned 


19.7 


16.7 


3.0 


2.0 


17.9 


19.5 


-1.6 


1.5 


t 


Regular credits 


14.5 


11.4 


3.1 * 


1.7 


12.1 


13.3 


-1.2 


1.3 


t 


Equated credits 


5.2 


5.3 


-0.1 


0.6 


5.8 


6.2 


-0.4 


0.6 




Passed Developmental Math (%) 


48.8 


44.9 


3.9 


5.6 


48.9 


48.9 


0.0 


3.9 




Passed Developmental English (%) 


28.3 


32.3 


-4.0 


5.5 


32.7 


33.1 


-0.4 


4.3 




Passed Developmental Reading (%) 


28.8 


24.5 


4.4 


4.9 


29.3 


33.2 


-3.9 


4.3 




Term GPA (%) 




















No GPA'’ 


28.2 


24.3 


3.8 


5.3 


29.1 


24.9 


4.2 


4.0 




3.0 to 4.0 or B/A 


20.6 


18.4 


2.3 


4.5 


20.5 


21.8 


-1.3 


3.5 




2.0 to 2.9 or C/B- 


21.4 


24.3 


-2.9 


4.9 


18.3 


21.5 


-3.2 


3.3 




1.0 to 1.9orD/C- 


13.1 


13.9 


-0.8 


4.5 


15.0 


14.8 


0.2 


2.9 




0 to 0.9 or E/D- 


16.8 


19.1 


-2.3 


4.3 


17.1 


17.0 


0.1 


3.1 




Sample size (total = 911) 


131 


136 






327 


317 









(continued) 






Table 4.7 (continued) 



SOURCE: MDRC calculations from Guilford Technical Community College transcript data . 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 
percent; * = 10 percent. 

A two-tailed t-test was applied to differences of impacts between subgroups. Statistical significance levels are indicated as: |tt = 1 percent; tt = 5 
percent; t = 1 0 percent. 

Standard errors are clustered by the Study Skills Course (ACA 118) section for the program group. 

“Students who dropped all courses before the drop date are not counted as registered. 

'’The "No GPA" category includes students who withdrew from all classes and students who took only developmental courses, which are not 
included in GPA calculations. The Study Skills course (ACA 118) was not included in GPA calculations for this table. 
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instructors had a higher level of enthusiasm about the success course and more extensive 
support and training during the first program semester than in later semesters. It is possible that 
this elevated level of interest and support may have resulted in the first cohort of students 
receiving a stronger version of the student success course, which could have resulted in more 
substantial impacts. Additionally, later instmctors’ lower levels of excitement may have 
dampened students’ achievement. However, some of the differences seem to be related to 
control group students performing better in later cohorts than in the first cohort. Since the 
cohorts are similar in background characteristics, there may be other explanations for the 
differences between the cohorts. As a result, it is difficult to make a definitive conclusion about 
the causes of these differential effects.'^ 



^The survey outcomes, when analyzed by cohort, were consistent with these findings, but the differential 
impacts between cohorts were not statistically significant for the survey, likely because of the smaller survey 
respondent sample size. See Appendix Table C.l for details. 
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Chapter 5 

Summary and Conclusions 



When Guilford sought to rigorously examine the effects of its new student success course, 
administrators and faculty alike hoped to learn whether the course could improve developmen- 
tal education students’ academic achievement and, if so, how these effects came about. In 
particular, Guilford’s leaders hoped that this course would foster the kinds of social and emo- 
tional behaviors that would allow their developmental education students to make an easier 
transition to college life. Additionally, with the newfound skills and the additional academic 
preparation the course offered, these leaders hoped that the developmental education students 
who took the course would be more motivated and better prepared to continue in college, earn 
higher grades and more credits, and ultimately earn a degree. 

The results of this study reveal that Guilford’s new success course, with its focus on so- 
cial and emotional skills, was able to foster some changes in students’ attitudes and perspec- 
tives. Specifically, the course was able to influence students’ emotional awareness of them- 
selves and others, their interest in learning, and their engagement in college. Such changes 
should not be taken lightly, as students’ interest in and commitment to learning is a foundational 
aspect of their continued persistence in college, particularly in the face of tough personal, 
economic, or academic challenges. Similarly, the course’s ability to foster certain “soft skills,’’ 
such as an awareness of others’ emotions and the ability to work productively in groups, is a 
key asset, as such skills are critical attributes for success in today’s 21st-century workplace.' 
The development of such skills alone may warrant colleges’ implementation of similar types of 
student success courses, which focus on building up students’ relational skills and personal 
awareness. 

However, although Guilford’s student success course was able to foster a number of 
positive personal habits and perspectives in students, these attributes did not translate into 
improved academic outcomes for the overall group of students to whom it was offered. That 
said, the students in the first cohort of the study did show some substantial academic improve- 
ments, so the lack of overall impacts may have resulted from the differential ways in which the 
course was offered across these three semesters. The earliest cohort of students may have 
received a qualitatively different success course than those who enrolled in later semesters. It is 
possible that the higher level of enthusiasm and support of instmctors in the first cohort may 
have influenced the effectiveness of the course. 



'Mumane and Levy (1996). 
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The quality of a program’s implementation can have important effects on students’ 
achievement. For example, an evaluation of a program designed to improve former Temporary 
Assistance for Needy Families recipients’ employment outcomes by providing job preparation 
and placement, work stabilization, and advancement services in six South Carolina counties 
found that one county did significantly better than the others in increasing the number of 
participants who obtained employment. Use of retention and advancement services was higher 
in this county than the others. This was attributed to the fact that the implementation of the 
program in this county most closely mirrored what the program’s designers had envisioned.^ 

The differential impacts of the success course on the various cohorts could also speak to 
the challenges community colleges face in scaling up promising programs. The first cohort of 
students who enrolled in the success course were taught primarily by instructors who had 
attended a training led by the On Course creator and were enthusiastically spearheading the 
implementation of the new course. However, in order to support the continued scale-up of this 
intervention, the college chose to implement a train-the-trainer model, whereby Guilford faculty 
recmited and trained new teachers. Less support was provided to these new instmctors over the 
course of the semester, as the college focused on adding more sections. 

The differential findings among the three cohorts of students enrolled in this study sug- 
gest that the quality of the program’s implementation may have played a key role in students’ 
outcomes. It appears that the students who received a stronger version of the program in the first 
program semester may have benefitted more from the success course than those who enrolled in 
the course in later semesters. Studies of other community college programs, such as learning 
communities^ and a success program for probationary students,"* suggest a similar trend, as 
students who received stronger versions of the program had more promising academic results 
than those who received weaker versions. These findings thus underscore the need for program 
developers to pay careful attention to the development and implementation of their intervention 
as well as the staff charged with leading it. As Guilford’s results suggest, implementing a 
curriculum with enthusiastic instructors who have sufficient support could make an important 
difference in students’ academic outcomes. 

While, overall, the mixed results from this study do not provide strong evidence that a 
success course alone improves student academic outcomes, it should be remembered that this 
study is a test of a single success course, and the results should not be seen as definitive proof 
that no success course can increase students’ achievement. Indeed, these findings should be 
taken in concert with other studies that have shown more positive results for these courses. 



^Scrivener, Azurdia, and Page (2005). 
^Weiss, Visher, and Wathington (2010). 
“^Scrivener, Sommo, and Collado (2009). 
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particularly when success courses are one component in more comprehensive programs. For 
instance, the findings in this study are not unlike those from Chaffey College’s success course, 
which had no meaningful impacts on students’ academic outcomes in the program semester or 
first postprogram semester.^ However, an enhanced version of that program did have positive 
impacts on the number of credits students earned and on grade point averages, and helped a 
substantial proportion of students move off academic probation. 

Similarly, a number of positive impacts on students’ outcomes have been observed in 
learning communities that included a student success course. For example, a learning communi- 
ties program at Houston Community College linked student success courses with developmen- 
tal math courses and also provided enhanced student services. Students who participated in this 
program were more likely to pass their developmental math course than those who attempted 
the same course in stand-alone classes. These results could be seen as an effect of students 
having learned study skills in the student success course.® A learning communities program at 
Kingsborough Community College in Brooklyn, New York, which linked a student success 
course with English and content courses and provided additional enhanced services for incom- 
ing freshmen, had some similar effects. Students who participated in the program moved more 
quickly through developmental English requirements than their control group counterparts and 
passed standardized reading and writing skills tests (needed to transfer to a four-year college) at 
a higher rate both during the program semester and by the end of a two-year follow-up period.’ 
These findings reveal that student success courses, along with other services, can positively 
influence students’ outcomes. 

However, even when program implementation is strong, it should also be noted that the 
impact on students’ achievement of a one-semester intervention may be modest and less likely 
to achieve the lofty goals of improved grades, credits earned, and graduation across the board. 
Eor instance, in studies of learning communities, which linked developmental or credit-bearing 
courses with a student success course for one semester, the impacts on students’ academic 
achievement tended to be modest and generally only sustained during the semester in which the 
program operated.* Similar results have also been seen for one- and two-semester advising 
interventions and mentoring programs as well programs that included student success courses.'^ 

These studies reveal that though programs that focus on improving particular aspects of 
students’ college experience are worthwhile, such as a success course focused on students’ well- 
being or more intensive advising programs, they may not be sufficient to translate into greater 

® Scrivener, Sommo, and Collado (2009). 

®Weissman et al. (201 1). 

’Scrivener et al. (2008). 

*Weissman et al. (201 1); Weiss, Visher, and Wathington (2010). 

^Scrivener and Coghlan (201 1); Visher, Butcher, and Cema (2010). 
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academic success. The short-term effects on student outcomes that resulted from these programs 
suggest that other issues may be creating more substantial barriers to students’ academic 
progress. For instance, while success courses may improve students’ soft skills and help them 
better understand how to manage college life, students may still be overwhelmed by larger 
challenges in their lives, such as an inability to afford their school tuition or the need to balance 
work, school, and family responsibilities. Similarly, success courses, and other, similar student 
support interventions, do not offer solutions to some of the sttuctural or academic barriers 
students may face in college, such as having to pass a long sequence of developmental educa- 
tion courses or continued failure in a particular academic course. 

Given these issues, colleges should carefully consider how success courses are integrat- 
ed into the larger fabric of students’ course work. Success courses may play an important role in 
helping introduce and support students as they make the transition to college, allowing them to 
learn about the variety of programs and services that may benefit them during their academic 
career. However, these courses also present an opportunity cost for students, as they often enroll 
in a success course at the expense of an academic course. As this study found, students earned 
fewer academic credits during the semester that they took the success course, in part because 
they were unable to fit in an additional academic course. An additional consideration is that 
most success courses do not offer credits that are transferrable to four-year institutions. 

With these concerns in mind, colleges may wish to seek ways to incorporate success 
courses into larger, more systemic approaches to improving developmental education students’ 
academic experiences. Given the relatively short-term effects of these courses and the oppor- 
tunity costs they present to students, colleges may need to consider smaller interventions that 
support students’ psychosocial well-being but do not require students to choose between their 
academic course-taking and their introduction to college life. In order to effect greater changes 
in students’ achievement, colleges might look toward more comprehensive approaches for 
improving developmental education students’ academic performance, such as reforms in 
developmental education instmction or the structuring of developmental education course 
sequences. A different combination of these efforts, which addresses students’ social challenges 
while also focusing more concretely on their academic needs, may prove even more promising 
in fostering students’ academic progress. 
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Appendix A 



Technical Appendix 




Description of Survey Scaies 

The following multi-item scale measures are presented in Table 4.1 and Appendix Table C.l 
and were created using data from the student survey. Multi-item scales are useful for measur- 
ing complex constructs, such as those outlined below, because such constructs cannot be 
assessed easily within a single-item measure. The original source of these scales is the On 
Course self-assessment included in the curriculum.' For all measures, a summary scale score is 
calculated and then divided by the number of items that make up the scale, to create an average 
scale score. Finally, Cronbach’s alpha^ — an indicator of how well the items included in the 
scale measure a common underlying construct — is presented for each scale. Higher values 
indicate that the scale is representative of an underlying construct: A Cronbach’s alpha of 0.8 
or above indicates good internal consistency of the scale, with slightly lower values being 
considered acceptable. 

Accepting Personal Responsibility (5-item scale, Cronbach’s Alpha = 0.75) 

1. 1 have become more aware of my responsibility for my academic learning. 

2. 1 now make more of an effort to attend class on time. 

3. 1 now make more of an effort to attend class regularly. 

4. 1 am more likely to find positive ways to solve my problems. 

5. If I miss class, I am more likely to find out what assignments I missed. 

Self-Motivation (3-item scale, Cronbach’s Alpha = 0.80) 

1. 1 have developed higher expectations for my future. 

2. Receiving a college education has become a high priority in my life. 

3. 1 have learned how to better plan for my educational goals. 

Self-Management (4-item scale, Cronbach’s Alpha = 0.76) 

1. 1 know more about how to use organizational tools such as calendars or to-do lists. 

2. 1 am better able to prioritize tasks, so that I give myself enough time to get things done. 



'Downing (2008) was used, with the exception of five questions that were phrased negatively and that 
loaded heavily on their own latent construct in a factor analysis, making them bad fits for the scales used here. 
^Cronbach (1951). 



51 




3. 1 have become more committed to my schoolwork. 

4. 1 have a better understanding of how my schoolwork affects my success. 

Interdependence (5-item scale, Cron bach’s Alpha = 0.80) 

1. 1 have learned how to use study groups more effectively. 

2. 1 have found a trustworthy person to turn to if I have problems. 

3. 1 have learned how to listen more carefully to people. 

4. 1 am more aware of where to go for help when I am confused about which courses to take. 

5. 1 now know where to go to get help if I am having trouble paying my tuition. 

Self-Awareness (4-item scale, Cronbach’s Alpha = 0.78) 

1. 1 have become more aware of my strengths and weaknesses. 

2. 1 have a greater ability to change bad habits. 

3. 1 am more aware of the ways I can get off track from achieving my goals and dreams. 

4. 1 am more likely to succeed in college. 

Lifelong Learning (4-item scale, Cronbach’s Alpha = 0.82) 

1. 1 have developed better study skills. 

2. 1 have a better understanding of how I learn. 

3. 1 have discovered how to learn from my failures and disappointments. 

4. 1 have figured out how to learn a subject even if it is hard to learn in class. 

Emotional Intelligence (4-item scale, Cronbach’s Alpha = 0.79) 

1. 1 have better control over my emotions, such as anger or sadness. 

2. 1 have become more aware of my feelings. 

3. 1 have become more aware of the feelings of others. 

4. Choosing courses in school that are both challenging and interesting has become more 
important to me. 
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Belief in Self (5-item scale, Cronbach’s Alpha = 0.86) 

1. 1 have developed a greater sense of confidence in my academic abilities. 

2. 1 feel better about who I am. 

3. 1 have learned how to celebrate my accomplishments. 

4. 1 have learned that high self-esteem can help me succeed. 

5. 1 am more honest with others about my needs. 

Response categories for all of the above questions: 

Strongly disagree (1) 

Disagree (2) 

Neutral (3) 

Agree (4) 

Strongly agree (5) 

Responses were summed and averaged within scales. Scores ranged from 1 to 5 within all the 
above scales. 

Use of Guilford Services (9-item scale, Cronbach’s Alpha = 0.79) 

How often have you used... 

1. Academic advising/planning 

2. Career Center 

3. Counseling 

4. Tutoring Center 

5. Walk-in Math Lab 

6. Transitions Program 

7. Mentoring 

8. College Transfer Center 

9. Writing Lab 
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Response categories: 

Rarely/Never (1) 

Sometimes (2) 

Often (3) 

Responses were summed and averaged. Scores ranged from 1 to 3. 

Positive Engagement (17-item scale, Cronbach’s Alpha = 0.90) 

1. Asked questions in class or contributed to class discussions 

2. Made a class presentation 

3. Prepared two or more drafts of a paper or assignment before turning it in 

4. Worked on a paper or project that required integrating ideas or information from 

various sources 

5. Worked with other students on projects during class 

6. Worked with classmates outside of class to prepare class assignments 

7. Used the Internet or instant messaging to work on an assignment 

8. Used e-mail to communicate with an instructor 

9. Discussed grades or assignments with an instructor 

10. Talked about career plans with an instructor or advisor 

11. Discussed ideas from your readings or classes with instructors outside of class 

12. Received prompt feedback (written or oral) from instructors on your performance 

13. Worked harder than you thought you could to meet an instmctor's standards 

or expectations 

14. Worked with instructors on activities other than course work 

15. Discussed ideas from your readings or classes with others outside of class (students, 

family members, coworkers, etc.) 

16. Had meaningful conversations with students of a different race or ethnicity other than 

your own 

17. Had meaningful conversations with students who differ Ifom you in terms of their 

religious beliefs, political opinions, or personal values 

Response categories; 

Never (1) 

Sometimes (2) 

Often (3) 

Very Often (4) 

Responses were summed and averaged. Scores ranged from 1 to 4. 
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Appendix B 

Sensitivity Anaiysis 




As discussed in Chapter 4, a substantial proportion of students assigned to the program group 
did not remain enrolled in the success course after the add/drop period. In order to assess 
whether this is the main reason for a lack of academic impacts, a nonexperimental analysis is 
presented here on program semester students who actually attended Guilford that semester. 

An analysis of program group students revealed that half of those who did not take the 
success course did not take any classes in the program semester at all. Removing the propor- 
tions of program and control group students who took no classes during the program semester 
provides a relatively comparable group of students among these two groups. By dropping 
nonenrollees from both groups, the program group achieves a relatively higher participation rate 
in the success course (83 percent of the program group as opposed to 72 percent in the full 
sample), allowing for a relatively unbiased estimate of student achievement with a higher 
participation rate. 

This exploratory analysis suggests some minor changes in students’ achievement. 
While most academic outcomes did not change in this subanalysis, the success course was 
associated with a decrease in students’ completion of developmental English during the pro- 
gram semester (see Appendix Table B.l).^ However, during the first postprogram semester, the 
student success course was associated with an increase in students’ developmental math pass 
rates (see Appendix Table B.2). Nevertheless, the academic outcome impacts presented in these 
tables are substantively the same as those from the experimental analysis of the full study 
sample and do not offer evidence that increased participation alone would have led to improved 
academic outcomes for these students. 



'Appendix Table B.l also shows a marked increase in students with “No GPA,” an increase in the number 
of courses attempted, and a decrease in equated credits earned. However, these outcomes are driven by 
program students’ enrollment in the student success course and their higher participation rate in this subsample. 
Similar to the description provided in the earlier analysis of academic outcomes, program students taking the 
success course were less likely than the control group to have had the time or resources to take some other 
courses that might have counted toward their GPA or toward other non-GPA, equated-credit courses. Although 
the two-credit success course contributes fewer credits than a typical class, it does seem to have tipped the 
balance of the average number of courses attempted. 
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Achieving the Dream: Community Colleges Count 
Appendix Table B.l 



Academic Outcomes in Program Semester (Program Semester Enrollees Only) 
Guilford Technical Community College 



Outcome 


Program 

Group 


Control 

Group 


Difference 


Standard 

Error 


Registered for any courses 


100.0 


100.0 


0.0 


0.0 


Registered for study skills course (%) 


83.5 


3.6 


79 9 


3.7 


Number of courses attempted 


3.8 


3.5 


0 ^ 


0.1 


Number of credits attempted 


11.9 


11.9 


0.0 


0.3 


Regular credits 


5.6 


5.3 


0.2 


0.3 


Equated credits 


6.3 


6.5 


-0.2 


0.4 


Attempted Developmental Math (%) 


72.2 


73.0 


-0.8 


3.3 


Attempted Developmental English (%) 


43.7 


47.0 


-3.2 


4.7 


Attempted Developmental Reading (%) 


43.8 


45.7 


-1.9 


3.9 


Passed all courses (%) 


37.1 


42.3 


-5.2 


3.2 


Withdrew from any courses (%) 


41.3 


42.5 


-1.2 


3.9 


Number of credits earned 


7.2 


7.6 


-0.5 


0.4 


Regular credits 


3.5 


3.4 


0.1 


0.3 


Equated credits 


3.7 


4.2 


-0.5 * 


0.3 


Passed Developmental Math (%) 


40.9 


44.5 


-3.6 


4.1 


Passed Developmental English (%) 


25.1 


31.6 


-6.5 * 


3.5 


Passed Developmental Reading (%) 


26.6 


30.8 


-4.3 


3.4 


Term GPA (%) 


No GPA 


47.1 


40.4 


6.6 * 


3.8 


3.0 to 4.0 or B/A 


24.6 


26.0 


-1.4 


2.9 


2.0 to 2.9 or C/B- 


13.0 


14.5 


-1.5 


2.4 


1.0 to 1.9 or D/C- 


5.6 


6.8 


-1.2 


1.7 


0 to 0.9 or F/D- 


9.7 


12.3 


-2.5 


2.1 


Sample size (total = 774 ) 


393 


381 







SOURCE: MDRC calculations from Guilford Technical Community College transcript data. 



NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are 
indicated as: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted by campus and research cohort. 

Standard errors are clustered by Study Skills course (ACA 118) section for the program group. 

^Students who dropped all courses before the drop date are not counted as registered. 

'’The "No GPA" category includes students who withdrew from all classes and students who took only 
developmental courses, which are not included in GPA calculations. The Study Skills course (ACA 118) was not 
included in GPA calculations for this table. 
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Achieving the Dream: Community Colleges Count 
Appendix Table B,2 



Academic Outcomes in First Postprogram Semester 
(Program Semester Enrollees Only) 
Guilford Technical Community College 



Outcome 


Program 

Group 


Control 

Group 


Difference 


Standard 

Error 


First Dostoroaram semester 


Registered for any courses (%)“ 


71.6 


70.0 


1.6 


3.2 


Enrolled full time (%) 


53.0 


51.7 


1.3 


4.8 


Number of credits attempted 


8.6 


8.3 


0.3 


0.5 


Regular credits 


6.0 


5.7 


0.3 


0.5 


Equated credits 


2.6 


2.6 


0.0 


0.3 


Attempted Developmental Math (%) 


33.9 


30.1 


3.9 


3.3 


Attempted Developmental English (%) 


18.1 


20.4 


-2.3 


2.7 


Attempted Developmental Reading (%) 


14.2 


15.0 


-0.8 


2.7 


Number of credits earned 


5.0 


4.9 


0.2 


0.4 


Regular credits 


3.7 


3.6 


0.0 


0.4 


Equated credits 


1.4 


1.3 


0.1 


0.2 


Passed Developmental Math (%) 


17.6 


13.6 


4.0 * 


2.4 


Passed Developmental English (%) 


9.8 


10.1 


-0.3 


1.8 


Passed Developmental Reading (%) 


7.8 


8.0 


-0.1 


1.9 


Term GPA (%) 


No GPA 


44.7 


45.5 


-0.9 


3.2 


3.0 to 4.0 or B/A 


22.6 


24.2 


-1.6 


3.1 


2.0 to 2.9 or C/B- 


11.3 


13.8 


-2.5 


2.2 


1.0 to 1.9 orD/C- 


8.0 


5.4 


2.5 


2.0 


0 to 0.9 or E/D- 


13.5 


11.0 


2.4 


2.0 


Sample size (total =774 ) 


393 


381 







SOURCE; MDRC calculations from Guilford Technical Community College transcript data. 



NOTES; Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance 
levels are indicated as; *** = 1 percent; ** = 5 percent; * = 10 percent. 

Estimates are adjusted hy campus and research cohort. 

Standard errors are clustered hy the Study Skills course (ACA 118) section for the program group. 

“Students who dropped all courses before the drop date are not counted as registered. 

'’The "No GPA" category includes students who were not registered, students who withdrew from 
all classes, and students who took only developmental courses, which are not included in GPA 
calculations. 
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Appendix C 

Survey Outcomes by Cohort 




Achieving the Dream: Community Colleges Count 
Appendix Table C.l 



Survey Outeomes, by Cohort 
Guilford Teebnieal Community College 



Outcome 


Spring 2008 Cohort 
Program Control 
Group Group Difference 


Standard 

Error 


Eall 2008 and Spring 2009 Cohorts 
Program Control Standard 

Group Group Difference Error 


Difference 

Between 

Subgroups 


Accepting personal responsibility** 




















Low 


7.2 


11.3 


-4.0 


5.1 


17.1 


12.9 


4.3 


2.9 




High 


27.5 


25.1 


2.3 


6.4 


26.6 


25.6 


0.9 


3.3 




Self-motivation’’ 




















Low 


10.4 


16.2 


-5.8 


5.3 


23.9 


24.9 


-1.0 


3.5 




High 


0.0 


0.0 


0.0 


0.0 


0.0 


0.0 


0.0 


0.0 




Self-managemenL 




















Low 


5.3 


14.1 


-8.8 ** 


3.9 


17.4 


22.4 


-5.0 


3.3 




High 


31.6 


30.2 


1.4 


6.0 


25.4 


20.4 


4.9 


3.8 




Interdependence*’ 




















Low 


4.1 


13.2 


-9.1 ** 


4.1 


13.2 


20.2 


-7.0 * 


3.7 




High 


27.2 


20.4 


6.8 


5.7 


19.8 


15.7 


4.1 


3.8 




Self-awareness® 




















Low 


6.2 


16.3 


-10.1 ** 


4.7 


15.3 


18.9 


-3.6 


3.5 




High 


29.2 


24.5 


4.6 


5.6 


26.1 


23.5 


2.6 


3.7 




Lifelong learning’ 




















Low 


2.9 


12.3 


_9 4 ** 


4.0 


10.7 


14.6 


-4.0 


3.1 




High 


29.5 


25.2 


4.4 


5.7 


22.7 


23.5 


-0.8 


4.0 




Emotional intelligence® 




















Low 


2.9 


14.4 


-11.5 *** 


4.0 


9.4 


17.2 


_7 9 ** 


3.7 




High 


25.0 


26.5 


-1.6 


5.6 


16.3 


16.6 


-0.3 


3.1 





(continued) 




Appendix Table C.l (continued) 







Spring 2008 Cohort 




1 Fall 2008 and Spring 2009 Cohorts I 


Difference 


Outcome 


Program Control 
Group Group Difference 


Standard 

Error 


Program Control 
Group Group Difference 


Standard 

Error 


Between 

Subgroups 


Belief in self*’ 




















Low 


0.9 


9.3 


-8.4 ** 


3.3 


10.7 


15.9 


-5.2 


3.4 




High 


34.9 


31.2 


3.7 


5.9 


23.1 


23.1 


0.0 


3.8 




Use of GTCC services' 




















Low 


8.2 


12.3 


-4.1 


4.2 


13.0 


16.8 


-3.8 


3.7 




High 


20.1 


19.1 


0.9 


6.1 


15.1 


14.2 


1.0 


4.0 




Positive engagement’ 




















Low 


11.2 


21.7 


-10.5 * 


5.6 


10.6 


16.4 


-5.8 * 


3.1 




High 


17.9 


16.3 


1.6 


5.1 


12.9 


15.3 


-2.4 


3.0 




Sample size (total = 661) 


95 


99 






234 


233 









SOURCE: MDRC calculations from Guilford Technical Community College survey data. 

NOTES: Rounding may cause slight discrepancies in sums and differences. 

A two-tailed t-test was applied to differences between research groups. Statistical significance levels are indicated as: *** = 1 percent; ** = 5 percent 
* = 10 percent. 

Estimates are adjusted by campus and research cohort. 

Standard errors are clustered by the Study Skills course (ACA 118) section for the program group. 

"Low" is the percentage of sample members scoring one standard deviation below the mean; "high" is the percentage of sample members scoring one 
standard deviation above the mean. 

®5-item scale about feelings of personal responsibility; response categories range from 1 = "strongly disagree" to 5 = "strongly agree." Items are 
averaged. 

’’3-item scale about feelings of self-motivation; response categories range from 1 = "strongly disagree" to 5 = "strongly agree." Items are averaged. 
‘’4-item scale about feelings of self-management; response categories range from 1 = "strongly disagree" to 5 = "strongly agree." Items are averaged. 
‘’5-item scale about feelings of interdependence; response categories range from 1 = "strongly disagree" to 5 = "strongly agree." Items are averaged. 
‘’4-item scale about feelings of self-awareness; response categories range from 1 = "strongly disagree" to 5 = "strongly agree." Items are averaged. 
4-item scale about feelings of lifelong learning; response categories range from 1 = "strongly disagree" to 5 = "strongly agree." Items are averaged. 
&4-item scale about feelings of emotional intelligence; response categories range from 1 = "strongly disagree" to 5 = "strongly agree." Items are 
averaged. 

’’5-item scale about feelings of belief in self; response categories range from 1 = "strongly disagree" to 5 = "strongly agree." Items are averaged. 
‘9-item scale about frequency of use of GTCC services; response categories range from 1 = "rarely/never" to 3 = "often." Items are averaged, 
j 17-item scale about frequencies of positive student engagement; response categories range from 1 = "never" to 4 = "very often." Items are averaged. 
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